BDD in agile testing: An experimental study

KỶ YẾU HỘI THẢO KHOA HỌC QUỐC GIA CITA 2020 “CNTT VÀ ỨNG DỤNG TRONG CÁC LĨNH VỰC” BDD in Agile Testing: An Experimental Study Le Thi Bich Tra Vietnam-Korea University of Information and Communication Technology The University of Da Nang ltbtra@vku.udn.vn Abstract. As a principle of project delivery, Agile has always been about achieving better outcomes faster through the use of a dynamic, collaborative framework than traditional methodologies. Behaviour Driven development (BDD) offers a modern agile software development approach which is seen as an evolution of Test Driven Development (TDD) and Acceptance Test Driven Development (ATDD). It introduces new methods and strategies that describes user behaviours by using an ubiquitous language (Gherkin) which are shared understanding of requirements between everyone involved in the software projects. This helps to improve communication and collaboration in the team lead to deliver higher quality software. However, it is unclear to why BDD enforces better quality software than TDD and ATDD to achieve the business goals and how to automatic step definitions from the features in Gherkin. In this paper it is examined the use of BDD in Agile Testing and compared with TDD and ATDD. I also conduct an example experiment to describe a domain-specific language and make acceptance test cases can be executable using Behave framework. Keywords: BDD, TDD, ATDD, Behave, Gherkin, Agile Testing. 1 Introduction In recent years, many companies have adopted Agile methodologies for their software development activity. The Agile concept which is defined in the Agile Manifesto and Twelve principles of Agile Software [7]. Many software companies use these principles to deliver value and quality projects to customers. In fact, VersionOne, the 14 annual State of Agile survey, a look into the global state of enterprise Agile in 2020 [27], reported that 95% of organizations practice Agile development. The top benefits realized by companies are reported such as: ability to manage changing priorities, project visibility, business/IT alignment, delivery speed/time to market, team morale, increase team productivity. In Agile approach, testing can begin at the start of the project with continuous integration between development and testing. It involves testing as early as possible and often as code becomes available and stable enough in the software development life cycle. The Test-First approach allows the team to focus on and clarify the expressed needs through a discussion of how to test the resulting code. The approach brings some benefits such as better design and code quality, faster time to market, up-to-date documentation, high code coverage, etc, ... [12, 9]. TDD, ATDD and BDD are common approaches in Agile software development which use Test First strategy test. TDD is more a development technique than a test technique. A developer may perform low level testing using the TDD, while a tester or product owner performs high level testing using ATDD. BDD can include both low level and high level tests. BDD use a simple domain specific language which is like natural language constructs that can express the behaviour and the expected outcomes [6]. The remainder of this is structured as follows: Section 2 presents the overview of BDD workflow. Section 3 explains why BDD has been adopted in Agile Testing compared with other TDD and ATDD approaches. Section 4 presents a sample experiment using Selenium WebDriver with Behave framework to demonstrate the code generation from scenarios apply BDD. Section 5 concludes and identifies future work. 130 Le Thi Bich Tra 2 Behavior Driven Development Behaviour Driven Development is a Test-First, Agile Testing practice, it was originally introduced by Dan North in 2003 that focuses on the behavior of the application from a business user’s perspective using concrete examples [16, 21]. BDD is a collaborative process that creates a shared understanding of requirements between the business and the Agile Team [5]. Fig. 1. BDD workflow As illustrated in Fig. 1, Agile team practicing BDD (the business analysts, developers, testers) come together, define requirements to achieve their business goals through features which are broken into smaller user stories from the perspective of the customer. Many teams write a short outline of the requirement in user story with format like this: As a I want So that The template is guidance to help teams capture enough information in the story, so that the conversation they have is valuable [20]. The feature can includes some scenarios with concrete examples of how we want the software to behave [21, 25]. The Agile team then writes down these examples in a domain specific language, such as Gherkin, created explicitly for writing behavior descriptions. This language follows a specific syntax, which uses the Given, When, Then keywords to describe what should happen when in the application. It is very similar to natural language, so that anyone can read and understand what it is testing. Then, some automation tools are used to execute the created specifications for testing [12, 16, 21, 22]. Scenarios are serves as a specification and are automated, so it can be called “Executable Specifications”. The Executable Specifications specify how the software should behave, they guide the developers in building working software that focuses on features that really matter to the business, and the tester uses the results of these tests as the starting point for manual and exploratory tests. As features are delivered, an up- to-date living documentation is produced [21, 15]. BDD practitioners express high-level requirements in the form of executable specifications. After Executable Specifications are implemented, at low-level specifications, the code is written in the step definitions is illustrated as unit-tests to help design and verify the application code. This approach of using unit tests to drive low-level design is a core TDD practice: describe the expected behavior as a failing test, write just enough code to get the test to pass, then refactor to avoid duplication and keep the code quality high. The application code to make the unit tests pass, repeat the process until acceptance criteria pass. 3 Why adopting BDD in Agile Testing? The 2015 edition of the Standish Group’s annual CHAOS Report found that 52% of projects were delivered late, ran over budget, or failed to deliver features for customer’s satisfaction and 19% of projects were canceled at some point during the development cycle [13]. There are many factors that may lead to the success or failure 131 KỶ YẾU HỘI THẢO KHOA HỌC QUỐC GIA CITA 2020 “CNTT VÀ ỨNG DỤNG TRONG CÁC LĨNH VỰC” of a software project: unrealistic goals, inaccurate estimates of needed resources, badly defined system requirements... One of the most common problem in software projects is lack of communication among customers, developers, and users [11, 18]. Mostly, developers do not quite understand what needs to be done, and the business people misunderstand what are the capacities of the developers and the implementation difficulties of the software they ask for [21]. Due to this, there are many opportunities for information to get lost in translation, be misunderstood, or just be ignored leading to business software failure. BDD was as a solution to this poor communication problems. BDD is an enhancement of The Three Amigos (representing the developer, the tester and the business) which involves constant collaboration and help teams overcome misinterpreted requirements and delivery issues [10, 14, 23]. When adopting BDD it is important to focus on solving the problems of delivering the software customers want and not only in testing automation techniques. BDD helps developers know what to test, where to start and how to name their tests and why tests failed [21, 24]. The shift from TDD to BDD is subtle but significant. Instead of thinking in terms of verification of a unit of code, we think in terms of a specification of how that code should behave-what it should do. The word “test” was replaced by “should” in language testing. This helped to make the test method names more meaningful and enabled developers to focus on the behavior of the application or underlying business requirements [6]. In addition, one limitation with TDD is that if the developer misunderstands what the software is to do, the unit tests will also include the same misunderstandings, giving passing results even though the software is not working properly [7]. For developers, a key advantage of BDD is that they do not have to rewrite tests each time they rewrite their code. That’s because BDD is not focused on the code but on the behavior of the application. In TDD, the tests written are typically more difficult to interpret by stakeholders, as they are highly technical and stakeholders may not have programming experience, un- like BDD, anyone involved can understand and process the test cases by using non-programmatic and ubiquitous language. In addition, communication is improved when compared to other ways of discussing requirements specifications with customers. BDD can be seen as an evolution of TDD and ATDD since it is an outside-in development approach [24]. TDD is focused towards building correct software but not on building the software that the user wants, developer focuses on covering low-level test as well as on the design of the interface and classes (unit level mostly). On the other hand, ATDD focuses on helping developers build the right software by promoting the collaborative construction of high-level acceptance tests (integration and acceptance testing). BDD leverages ATDD along with other techniques around conversations, collaboration and automation to en- sure that a team delivers what the business wants based on a common language where all stakeholders can understand. ATDD and TDD are methodologies used within BDD at a higher level. The benefits of the outside in development approach is that focusing on business value, encouraging well-designed code so easier to understand code and avoiding waste [21]. 4 Experimental and evaluation Steps definitions in scenarios are automated by BDD tools such as JBehave for Java, SpecFlow for .NET and Behave for Python, Cucumber for Ruby,...[3, 4, 1, 2]. Behave is one of the popular BDD frameworks that is preferred by Python practitioner and best-suited for automated browser testing with Selenium [19, 8]. To demonstrate how to automatic step definitions in BDD, I have conducted a sample example with an existing web system. The first stage is identifying identify acceptance criteria for user stories to deliver a login feature. The domain- specific language uses natural English language constructs to define the outcomes from the behavior in the feature. An expected behavior could be for the scenarios is expect whether login to the demo website successfully. The login feature with acceptance criteria are written in Gherkin format (Given, When and Then) can be seen in Figure 2. The scenario comprises basic three types of steps: [26, 12] - Given steps describe the pre-conditions for the scenario and prepares the test environments. - When steps describe the actions under test. - Then steps describe the post-conditions that should be satisfied if the system meets the specification. 132 Le Thi Bich Tra The Gherkin language also support Background allows to add some context to the scenarios. Scenario Outlines makes it possible to run the same scenario with different values defined in an example table. This is a simple but powerful way to expand test coverage without code duplication. Figure 3 demonstrates how the Python code mapped to each above step defined Fig. 2. Login feature example written in Gherkin in the scenarios using Behave framework. Each step in the feature is mapped to an acceptance test written in step implements. Five methods will be automatically generated from the scenario with their annotations, where the header text will match the description of the steps. In Behave these methods are convention- ally stored in a LoginSteps.py file within the test suite. Given -When -Then steps that only have to be implemented once and reused in many scenarios even different features, this not only helps to save amount of code but also contributes a lot of maintainability of the tests. The step parameters username and password are used to parameterize the test steps which helps to save a lot of time to update code implementation as you only need to update your feature files and your code will adapt on what you supplied between the double quotes. Parameterization can help a team adopt reusable set of steps, and it inspires healthier discussion when writing scenarios. The assert statement will raise a runtime-error if the condition in it results in False. The example use Selenium WebDriver which is a well-known package for web browser automation. The Page Objects Model (POM) design pattern helps to reusable code that can be shared across multiple test case and reducing the amount of duplicated code. With POM, several changes in user interface can be easily solved in only one place. Behave framework provides some test hooks such as fixtures –startup and 133 KỶ YẾU HỘI THẢO KHOA HỌC QUỐC GIA CITA 2020 “CNTT VÀ ỨNG DỤNG TRONG CÁC LĨNH VỰC” Fig. 3. Translating Test Steps in Login Feature File to Python Code clean up function, just like in TDD. These test hooks for Gherkin tests can be defined in an environment.py file which contains some pre-tests and post-tests instructions during the testing automation. These can be test-data preparations (fixtures), initializing web drivers for UI Tests or kicking off a mock server for component level testing [17]. Hooks can be associated with Features, Scenarios, Steps, Tags or the whole test run. As Figure 3 shows, hook before scenario to run before each Scenario to specify browser for testing and initialize web elements for each page by using Selenium. Testing the above example will produce a report by Allure which is an open- Fig. 4. Test hooks example in Behave source framework designed to create test execution reports that are clear to everyone in the team. Allure Report provides a nice big picture of which features have been covered, where defects are clustered, how the time of execution takes and so on for the team as well. In the example, writing scenarios from the beginning enforces quality-first and 134 Le Thi Bich Tra Fig. 5. Report with Allure-behave test-first mindsets. BDD automation can run scenarios to fail until the feature is implemented and causes tests to pass. Given -When -Then steps can be reused between scenarios and the underlying implementation for each step does not change. 5 Conclusion This paper has described a detailed workflow for BDD and explained why adopting it. Because BDD has encourages communication and coordination within the team, thus ensuring high quality right from the beginning of the project so that it is more effective than TDD and ATDD approach. I also evaluate some benefits when generating automatic step definitions from scenarios in Behave. In the future, I am planning to write more focused and concise unit tests on low-level in BDD style. In addition, applying Mocking function, by isolating dependencies, the tests run more quickly, more reliably and avoiding undesired side effects during testing. References 1. Behave, https://behave.readthedocs.io/en/latest/ 2. Cucumber, https://docs.cucumber.io/ 3. JBehave, https://jbehave.org/ 4. SpecFlow, https://specow.org 5. Agile, S.: Advanced topic - Behavior-Driven Development (Last accessed 20 September 2020), https://www.scaledagileframework.com/ behavior-driven-development/ 6. Ahonen, A., et al.: Unit and integration testing of java: Jvm behavior-driven development testing frameworks vs. junit (2017) 7. Black, R., Coleman, G., Walsh, M., Cornanguer, B., Forgacs, I., Kakkonen, K., Sabak, J., Black, R.: Agile testing foundations: An istqb foundation level agile tester guide. pp. 78–84. BCS (2017) 8. Bose, S.: https://www.browserstack.com/guide/top-python-testing-frameworks (Last accessed 20 September 2020), https://www.browserstack.com/guide/ top-python-testing-frameworks 9. Buchan, J., Li, L., MacDonell, S.G.: Causal factors, benefits and challenges of test- driven development: Practitioner perceptions. In: 2011 18th Asia-Pacific Software Engineering Conference. pp. 405–413. IEEE (2011) 10. Bussenot, R., Leblanc, H., Percebois, C.: Orchestration of domain specific test languages with a behavior driven development approach. In: 2018 13th Annual Conference on System of Systems Engineering (SoSE). pp. 431– 437. IEEE (2018) 11. Charette, R.N.: Why software fails [software failure]. IEEE spectrum 42(9), 42–49 (2005) 12. Farcic, V., Garcia, A.: Test-Driven Java Development: Invoke TDD principles for end-to-end application development. Packt Publishing Ltd (2018) 13. International, S.G.: The chaos report. United States of America (2015) 14. Kudryashov, K.: The beginner’s guide to bdd. Dan North Q & A. https://inviqa. com/blog/bdd-guide (2015) 15. Nicieja, K.: 3 Ways How Specification By Example and Gherkin Improve Col- laboration (Last accessed 20 September 2020), https://dzone.com/articles/ 3-ways-specification-by-example-with-gherkin-reduc 16. North, D.: Introducing BDD (Last accessed 20 September 2020), http:// dannorth.net/introducing-bdd/ 135 KỶ YẾU HỘI THẢO KHOA HỌC QUỐC GIA CITA 2020 “CNTT VÀ ỨNG DỤNG TRONG CÁC LĨNH VỰC” 17. Oropesa, K.: Behave Test Hooks - The Setup() and Teardown() Equivalent 18. Outsource2india: 10 Reasons that lead to software project failure (Last accessed 20 September 2020), https://www.outsource2india.com/software/SoftwareProjectFailure.asp 19. Panda, A.: Python BDD Framework Comparision (Last accessed 20. 20 September 2020), https://automationpanda.com/2019/04/02/ python-bdd-framework-comparison/ 21. Rose, S.: User stories and BDD (part 1)- the origins and evolution of the user story (Last accessed 20 September 2020), https://cucumber.io/blog/bdd/ user-stories-are-not-the-same-as-features/ 22. Smart, J.F.: BDD in Action, vol. 12. Manning Publications New York (2014) 23. Smartbear: TDD vs BDD vs ATDD - What’s the Right choice (Last accessed 20 September 2020), https://bitbar.com/blog/ tdd-vs-bdd-vs-atdd-whats-the-right-choice-for-you/ 24. Software, S.: Who does what (Last accessed 20 September 2020), https:// cucumber.io/docs/bdd/who-does- what/ 25. Solis, C., Wang, X.: A study of the characteristics of behaviour driven development. In: 2011 37th EUROMICRO Conference on Software Engineering and Advanced Applications. pp. 383–387. IEEE (2011) 26. SpecFlow: Getting Started with Behavior Driven Development (Last accessed 20 September 2020), https://specflow.org/bdd/ 27. Storer, T., Bob, R.: Behave nicely! Automatic generation of code for behaviour driven development test suites. In: 2019 19th International Working Conference on Source Code Analysis and Manipulation (SCAM). pp. 228– 237. IEEE (2019) 28. VersionOne: 14th annual state of agile report. pp. 8–10 (2020), https:// stateofagile.com/ 136