测试框架体系介绍

TDD DDT ATDD BDD MBT

ceshiren.com

测试框架是什么

A testing framework is a set of guidelines or rules used for creating and designing test cases. A framework is comprised of a combination of practices and tools that are designed to help QA professionals test more efficiently.
These guidelines could include coding standards, test-data handling methods, object repositories, processes for storing test results, or information on how to access external resources.

测试框架是一组用于创建和设计测试用例的指南或规则。框架由旨在帮助 QA 专业人员更有效地测试的实践和工具的组合组成。 这些指南可能包括编码标准、测试数据处理方法、对象存储库、存储测试结果的过程或有关如何访问外部资源的信息。

测试框架的价值

Testing frameworks are an essential part of any successful automated testing process. They can reduce maintenance costs and testing efforts and will provide a higher return on investment (ROI) for QA teams looking to optimize their agile processes.

测试框架是任何成功的自动化测试过程的重要组成部分。它们可以降低维护成本和测试工作，并为寻求优化其敏捷流程的 QA 团队提供更高的投资回报率 (ROI)。

测试框架的收益

  • Improved test efficiency 提高测试效率
  • Lower maintenance costs 降低维护成本
  • Minimal manual intervention 最少的人工干预
  • Maximum test coverage 最大测试覆盖率
  • Reusability of code 代码的可重用性

常见测试框架类型

框架 说明
TDD 代码风格
DDT 数据驱动风格
ATDD 验收测试驱动开发
BDD 行为驱动开发 (Behavior-driven development)
MBT Model Based Testing 基于模型的测试

TDD

TDD 定义

Test-driven development (TDD) is a software development process relying on software requirements being converted to test cases before software is fully developed, and tracking all software development by repeatedly testing the software against all test cases.

测试驱动开发（TDD）是一个软件开发过程，在软件完全开发之前，将软件需求转换为测试用例，并通过针对所有测试用例重复测试软件来跟踪所有软件开发。这与首先 ​​ 开发软件和稍后创建测试用例相反

TDD 流程

  • 单元测试
  • 重构
  • 覆盖率
  • 可测性提升
  • 模型驱动设计

TDD 来源于 XP 极限编程

代表作 JUnit TestNG

    @Test
    void standardAssertions() {
        assertEquals(2, calculator.add(1, 1));
        assertEquals(4, calculator.multiply(2, 2),
                "The optional failure message is now the last parameter");
        assertTrue('a' < 'b', () -> "Assertion messages can be lazily evaluated -- "
                + "to avoid constructing complex messages unnecessarily.");
    }

TestNG

package example1;

import org.testng.annotations.*;

public class SimpleTest {

 @BeforeClass
 public void setUp() {
   // code that will be invoked when this test is instantiated
 }

 @Test(groups = { "fast" })
 public void aFastTest() {
   System.out.println("Fast test");
 }

 @Test(groups = { "slow" })
 public void aSlowTest() {
    System.out.println("Slow test");
 }

}

代表作 Pytest UnitTest

# content of test_sample.py
def inc(x):
    return x + 1


def test_answer():
    assert inc(3) == 5

import unittest

class TestStringMethods(unittest.TestCase):

    def test_upper(self):
        self.assertEqual('foo'.upper(), 'FOO')

    def test_isupper(self):
        self.assertTrue('FOO'.isupper())
        self.assertFalse('Foo'.isupper())

    def test_split(self):
        s = 'hello world'
        self.assertEqual(s.split(), ['hello', 'world'])
        # check that s.split fails when the separator is not a string
        with self.assertRaises(TypeError):
            s.split(2)

BDD Behavior Driven Development

BDD

behavior-driven development (BDD) is an agile software development process that encourages collaboration among developers, quality assurance experts, and customer representatives in a software project. It encourages teams to use conversation and concrete examples to formalize a shared understanding of how the application should behave. It emerged from test-driven development (TDD). Behavior-driven development combines the general techniques and principles of TDD with ideas from domain-driven design and object-oriented analysis and design to provide software development and management teams with shared tools and a shared process to collaborate on software development.

在软件工程中，行为驱动开发 (BDD) 是一种敏捷软件开发过程，它鼓励软件项目中的开发人员、质量保证专家和客户代表之间进行协作。它鼓励团队使用对话和具体示例来形成对应用程序应该如何运行的共同理解。它源于测试驱动开发 (TDD)。行为驱动开发将 TDD 的通用技术和原则与领域驱动设计和对象的思想相结合面向分析和设计，为软件开发和管理团队提供共享工具和共享流程，以便在软件开发方面进行协作。

BDD VS TDD

BDD

BDD 与 TDD

BDD 相关框架

  • JBehave
  • Cucumber
  • Mspec
  • Specflow

Cucumber

Cucumber is a tool that supports Behaviour-Driven Development(BDD).

Cucumber 测试用例 Scenario 场景

Scenario: Finding some cheese
   Given I am on the Google search page
   When I search for "Cheese!"
   Then the page title should start with "cheese"

Cucumber 测试用例步骤定义

public class ExampleSteps {

    private final WebDriver driver = new FirefoxDriver();

    @Given("I am on the Google search page")
    public void I_visit_google() {
        driver.get("https://www.google.com");
    }

    @When("I search for {string}")
    public void search_for(String query) {
        WebElement element = driver.findElement(By.name("q"));
        // Enter something to search for
        element.sendKeys(query);
        // Now submit the form. WebDriver will find the form for us from the element
        element.submit();
   }

   @Then("the page title should start with {string}")
   public void checkTitle(String titleStartsWith) {
       // Google's search is rendered dynamically with JavaScript
       // Wait for the page to load timeout after ten seconds
       new WebDriverWait(driver,10L).until(new ExpectedCondition<Boolean>() {
           public Boolean apply(WebDriver d) {
               return d.getTitle().toLowerCase().startsWith(titleStartsWith);
           }
       });
   }

   @After()
   public void closeBrowser() {
       driver.quit();
   }
}

cucumber 项目结构

ATDD Acceptance Test Driven Development

ATDD 定义

Acceptance test–driven development (ATDD) is a development methodology based on communication between the business customers, the developers, and the testers. ATDD encompasses many of the same practices as specification by example (SBE), behavior-driven development (BDD), example-driven development (EDD), and support-driven development also called story test–driven development (SDD). All these processes aid developers and testers in understanding the customer’s needs prior to implementation and allow customers to be able to converse in their own domain language.

验收测试驱动开发 (ATDD) 是一种基于业务客户、开发人员和测试人员之间沟通的开发方法。ATDD 包含许多与示例规范 (SBE)、行为驱动开发 (BDD)、示例驱动开发 (EDD)、和支持驱动开发（也称为故事测试驱动开发（SDD）。所有这些流程都有助于开发人员和测试人员在实施之前了解客户的需求，并使客户能够使用他们自己的领域语言进行交流。

ATDD 相关工具

  • FitNesse：The fully integrated standalone wiki and acceptance testing framework
  • Robot Framework is a Python-based, extensible keyword-driven automation framework for acceptance testing, acceptance test driven development (ATDD), behavior driven development (BDD) and robotic process automation (RPA).

FitNesse

Robotframework

Robotframework 介绍

Robot Framework is a generic open source automation framework. It can be used for test automation and robotic process automation (RPA). Robot Framework has an easy syntax, utilizing human-readable keywords. Its capabilities can be extended by libraries implemented with Python, Java or many other programming languages. Robot Framework has a rich ecosystem around it, consisting of libraries and tools that are developed as separate projects.

Robotframework 测试用例

*** Settings ***
Documentation     Simple example using SeleniumLibrary.
Library           SeleniumLibrary

*** Variables ***
${LOGIN URL}      http://localhost:7272
${BROWSER}        Chrome

*** Test Cases ***
Valid Login
    Open Browser To Login Page
    Input Username    demo
    Input Password    mode
    Submit Credentials
    Welcome Page Should Be Open
    [Teardown]    Close Browser

*** Keywords ***
Open Browser To Login Page
    Open Browser    ${LOGIN URL}    ${BROWSER}
    Title Should Be    Login Page

Input Username
    [Arguments]    ${username}
    Input Text    username_field    ${username}

Input Password
    [Arguments]    ${password}
    Input Text    password_field    ${password}

Submit Credentials
    Click Button    login_button

Welcome Page Should Be Open
    Title Should Be    Welcome Page

数据驱动风格

*** Settings ***
Test Template    Login with invalid credentials should fail

*** Test Cases ***                USERNAME         PASSWORD
Invalid User Name                 invalid          ${VALID PASSWORD}
Invalid Password                  ${VALID USER}    invalid
Invalid User Name and Password    invalid          invalid
Empty User Name                   ${EMPTY}         ${VALID PASSWORD}
Empty Password                    ${VALID USER}    ${EMPTY}
Empty User Name and Password      ${EMPTY}         ${EMPTY}

BDD 风格

*** Test Cases ***
Valid Login
    Given login page is open
    When valid username and password are inserted
    and credentials are submitted
    Then welcome page should be open

ATDD VS BDD

TDD ATDD BDD
受众 开发 开发 测试 客户 开发 测试 客户
过程 代码 DSL 行为
目标 代码调用功能 验收测试 需求 需求

MBT Model Based Testing

Model-Based Testing GrapheWalker

  • model-based testing
  • GraphWalker, an open-source model-based testing tool

edge 代表步骤

An edge represents an action, a transition. An action could be an API call, a button click, a timeout, etc. Anything that moves your System Under Test into a new state that you want to verify. But remember, there is no verification going on in the edge. That happens only in the vertex.

一条边代表一个动作，一个过渡。 操作可以是 API 调用、按钮单击、超时等。任何将您的被测系统移动到您想要验证的新状态的任何事情。但请记住，边缘没有进行验证。这只发生在顶点。

vertex 代表断言

A vertex represents verification, an assertion. A verification is where you would have assertions in your code. It is here that you verify that an API call returns the correct values, that a button click actually did close a dialog, or that when the timeout should have occurred, the System Under Test triggered the expected event.

一个顶点代表验证，一个断言。 验证是您的代码中有断言的地方。在这里，您可以验证 API 调用是否返回正确的值、按钮单击是否确实关闭了对话框，或者在应该发生超时时，被测系统触发了预期的事件。

graph 代表测试用例集

A model is a graph, which is a set of vertices and edges From a model, GrapWalker will generate a path through it. A model has a start element, and a generator which rules how the path is generated, and associated stop condition which tells GraphWalker when to stop generating the path.

模型是一个图，它是一组顶点和边 从模型中，GrapWalker 将生成一条通过它的路径。一个模型有一个起始元素，一个规则如何生成路径的生成器，以及告诉 GraphWalker 何时停止生成路径的相关停止条件。

测试用例样板生成

@GraphWalker(value = "random(edge_coverage(100))")
public class OwnerInformationTest extends ExecutionContext implements OwnerInformation {

    private static final Logger log = LoggerFactory.getLogger(OwnerInformationTest.class);

    @Override
    public void v_OwnerInformation() {
        $(By.tagName("h2")).shouldHave(text("Owner Information"));
        setAttribute("numOfPets", Value.asValue($$x("//table/tbody/tr/td//dl").size()));
        log.info("Number of pets: " + getAttribute("numOfPets"));
    }

    @Override
    public void e_UpdatePet() {
        $("button[type=\"submit\"]").click();
    }

    @Override
    public void v_FindOwners() {
        $(By.tagName("h2")).shouldHave(text("Find Owners"));
        $(By.tagName("h2")).shouldBe(visible);
    }

    @Override
    public void e_EditPet() {
        $(By.linkText("Edit Pet")).click();
    }

    @Override
    public void e_AddNewPet() {
        $(By.linkText("Add New Pet")).click();
    }

    @Override
    public void e_AddVisit() {
        $(By.linkText("Add Visit")).click();
    }

    @Override
    public void e_FindOwners() {
        $("[title='find owners']").click();
    }

    @Override
    public void e_AddPetSuccessfully() {
        Date date = new Faker().date().past( 365 * 20, TimeUnit.DAYS);
        SimpleDateFormat sdf;
        sdf = new SimpleDateFormat("yyyy-MM-dd");
        String birthData = sdf.format(date);
        $(By.id("birthDate")).clear();
        $(By.id("birthDate")).sendKeys(birthData + Keys.ENTER);

        $(By.id("name")).clear();
        $(By.id("name")).sendKeys(new Faker().name().fullName());

        $(By.id("type")).selectOption(new Faker().number().numberBetween(0,5));
        $(By.cssSelector("button[type=\"submit\"]")).click();
    }

    @Override
    public void v_NewPet() {
        $(By.tagName("h2")).shouldHave(text("New Pet"));
        $(".has-feedback").shouldBe(visible);
    }

    @Override
    public void e_VisitAddedSuccessfully() {
        $(By.id("description")).clear();
        $(By.id("description")).sendKeys(new Faker().lorem().word());
        $("button[type=\"submit\"]").click();
    }

    @Override
    public void v_NewVisit() {
        $(By.tagName("h2")).shouldHave(text("New Visit"));
    }

    @Override
    public void v_Pet() {
        $(By.tagName("h2")).shouldHave(text("Pet"));
    }

    @Override
    public void e_AddPetFailed() {
        $(By.id("name")).clear();
        $(By.id("birthDate")).clear();
        $(By.id("birthDate")).sendKeys("2015/02/05" + Keys.ENTER);
        $(By.id("ui-datepicker-div")).shouldBe(not(visible));
        $(By.id("type")).selectOption("dog");
        $("button[type=\"submit\"]").click();
    }

    @Override
    public void e_VisitAddedFailed() {
        $(By.id("description")).clear();
        $("button[type=\"submit\"]").click();
    }
}

DDT Data Driven Testing

DDT

Data-driven testing (DDT), also known as table-driven testing or parameterized testing, is a software testing methodology that is used in the testing of computer software to describe testing done using a table of conditions directly as test inputs and verifiable outputs as well as the process where test environment settings and control are not hard-coded.

数据驱动测试（DDT），也称为表驱动测试或参数化测试，是一种软件测试方法，用于计算机软件的测试，用于描述使用条件表直接作为测试输入和可验证输出完成的测试以及测试环境设置和控制没有硬编码的过程

DDT 相关工具

  • DDT 是一种实践，可以跟很多框架结合
  • 单元测试结合 DDT：JUnit4 JUnit5 TestNG
  • RobotFramework DDT
  • YAML JSON CSV 驱动 HttpRunner

数据驱动应用案例

  • HttpRunner 可以根据代理抓包自动生成测试用例
  • YAPI、Swagger 等工具可以根据数据自动生成测试用例代码
  • JVM-Sandbox-Repeater Gor 录制工具可以把请求保存为测试用例并重放以实现快速回归测试

HttpRunner 测试框架

HttpRunner 测试用例

config:
  name: "request methods testcase with functions"
  variables:
    foo1: config_bar1
    foo2: config_bar2
    expect_foo1: config_bar1
    expect_foo2: config_bar2
  base_url: "https://postman-echo.com"
  verify: False
  export: ["foo3"]

teststeps:
  - name: get with params
    variables:
      foo1: bar11
      foo2: bar21
      sum_v: "${sum_two(1, 2)}"
    request:
      method: GET
      url: /get
      params:
        foo1: $foo1
        foo2: $foo2
        sum_v: $sum_v
      headers:
        User-Agent: HttpRunner/${get_httprunner_version()}
    extract:
      foo3: "body.args.foo2"
    validate:
      - eq: ["status_code", 200]
      - eq: ["body.args.foo1", "bar11"]
      - eq: ["body.args.sum_v", "3"]
      - eq: ["body.args.foo2", "bar21"]
  - name: post raw text
    variables:
      foo1: "bar12"
      foo3: "bar32"
    request:
      method: POST
      url: /post
      headers:
        User-Agent: HttpRunner/${get_httprunner_version()}
        Content-Type: "text/plain"
      data: "This is expected to be sent back as part of response body: $foo1-$foo2-$foo3."
    validate:
      - eq: ["status_code", 200]
      - eq:
          [
            "body.data",
            "This is expected to be sent back as part of response body: bar12-$expect_foo2-bar32.",
          ]
  - name: post form data
    variables:
      foo2: bar23
    request:
      method: POST
      url: /post
      headers:
        User-Agent: HttpRunner/${get_httprunner_version()}
        Content-Type: "application/x-www-form-urlencoded"
      data: "foo1=$foo1&foo2=$foo2&foo3=$foo3"
    validate:
      - eq: ["status_code", 200]
      - eq: ["body.form.foo1", "$expect_foo1"]
      - eq: ["body.form.foo2", "bar23"]
      - eq: ["body.form.foo3", "bar21"]

数据驱动风格为什么广受欢迎

维护成本最低，录制回放技术越来越成熟，可以与数据驱动很好的结合。低代码、用例生成技术的流行，会让数据驱动风格更受欢迎。

Q&A