This example applies Gotten, a generic model-driven engineering framework for metamorphic testing, to video streaming APIs.

Running example for video streaming APIs

This example demonstrates the generality and flexibility of Gotten by defining a metamorphic testing environment for video streaming APIs. The approach groups related APIs into domains, such as video streaming, audio streaming, hotel booking, and flight booking, as illustrated below.

Each domain can be described through a meta-model and a set of metamorphic relations. Different processors can then implement the corresponding API; for the video streaming domain, the processors considered here are YouTube and Vimeo.

The following figure presents the meta-model for video streaming APIs. The VideoAPITest class represents a named test case through its testName attribute. For simplicity, each test case contains one operation: search, upload, or update. Upload and update operations require a video parameter identified by an ID and may also include a title, a description, and a list of tags. A video can provide statistical data through VideoStatistics, including the number of views and likes. The results produced by each operation are stored through the results reference.

mrDSL program for video streaming APIs

Because it is difficult to define a conventional test oracle that determines whether a video streaming API behaves correctly, this example uses metamorphic testing. The following listing shows the mrDSL program used to apply Gotten to the video streaming API domain.

metamodel videostream "/video/model/VideoStream.ecore" with m1, m2
models "/video/model/videotc"

videostream input Features {
	context VideoAPITest def: IsFullSearch: Boolean =
          request.oclIsTypeOf(SearchVideo)
          and request.oclAsType(SearchVideo).maxResults = -1
  context VideoAPITest def: IsUpdate: Boolean = request.oclIsTypeOf(UpdateVideo)
  context SearchVideo def: MaxResults: Int = maxResults
  context SearchVideo def: SearchOrder: Int = orderType
  context SearchVideo def: UntilYear: Int = until.year
  context SearchVideo def: FromYear: Int = from.year
  context SearchVideo def: Radius: Double = position.radius
}

output Features {
  NVideos : Long
  Results: Set
  OutputVideoId: Long
  OutputVideoTitle: String
}

Processor {
  Name: String
  Version: String
}

MetamorphicRelations {
 MR1 = [ (IsFullSearch(m1) and SearchOrder(m1) <> SearchOrder(m2))
         implies (NVideos(m1) == NVideos(m2)) ]
 MR2 = [ (IsFullSearch(m1) and UntilYear(m1) < FromYear(m2))
         implies (Results(m1)->excludes(Results(m2))) ]
 MR3 = [ (IsFullSearch(m1) and Radius(m1) > Radius(m2))
         implies (Results(m1)->includes(Results(m2))) ]
 MR4 = [ (IsFullSearch(m1) and MaxResults(m1) >= MaxResults(m2))
         implies (NVideos(m1) >= NVideos(m2))]
 MR5 = [ (IsUpdate(m1) and m1 == m2)
         implies
         (OutputVideoId(m1) <> OutputVideoId(m2) and
         OutputVideoTitle(m1) == OutputVideoTitle(m2))]
}

Brief description of the metamorphic relations

The following table summarises the five metamorphic relations defined for the video streaming API example.

Experiment with YouTube and Vimeo

We conducted an experiment using MR1, MR4, and MR5. For MR1 and MR4, the source test cases included queries based on trending searches on YouTube and Vimeo, such as world cup and esports. These queries were complemented by boundary cases reported by Segura et al.¹, including winter pentathlon 1949 and mistrustfully. The same tests were reused for both platforms.

Executing all tests took 4 minutes and 4 seconds on YouTube and 10 minutes and 36 seconds on Vimeo. The results show that the framework can formalise the metamorphic relations described in natural language by Segura et al.¹, reproduce their results, and detect real issues reported in the issue-tracking systems of the platforms studied.

Download the YouTube and Vimeo project

The project archive includes the Gotten project for YouTube and Vimeo together with the data generated and used in the experiment described above.

• Gotten for YouTube and Vimeo

Video demonstration

Select the image below to watch a demonstration of the Gotten environment for video streaming APIs.

Acknowledgements

This work was funded by the Spanish Ministry of Science through project MASSIVE (RTI2018-095255-B-I00) and by the Madrid R&D programme through project FORTE (P2018/TCS-4314).