Custom JUnit5 Test Engines @Java.IL – the Israeli Java Community – 2020

hello again so this time in israel about
virtua
so i don’t know if any one of you attend
my i think it was functional director
from
the workshop we had did some some time
ago
so the most critical part now and so
really
the biggest challenge for me is now how
to share my screen
so i will try now um i hope that
i have the right for it already
then i will try so
and here we are so what i’m not doing is
uh going to this presentation
because then i’m losing the focus
between zoom and all this stuff again
and again so it’s it’s just too
complicated for me so
um i think you can see the slides um
[Music]
and yeah that’s it so
jonah five and then we want to talk a
little bit about custom test engines
um yeah as you heard already i’m working
for jfrog i’m sven
i’m a developer advocate for jfrog i
think about j i don’t have to say
too much
i think most people of you know it
already
and well if you have any questions or
whatever after the talk or
someday the easiest way to reach me is
definitely twitter
not email please don’t use my email
address
um what we want to do today
first of all i want to have a short j105
introduction
um i will explain why and what and all
that stuff don’t worry
then i want to show
how to test the test engine because if
you want to start
writing a test engine you should test
your test engine
uh that means you need to know how to
start a test engine
and after this we want to explore a
little bit what is the minimum
stuff you need to create a test engine
itself
um i will show you what is a
minimum part to define a test or what
what is the first step to
to start with your domain specific
language you want to have for
creating tests or whatever then how to
execute test plans and
some ideas what you can do and the last
part is more or less
what i saw so far in the field or what
i’ve done with test engines
and this really leads mostly into a
discussion about
the whole topic so if you have any
questions
i will have a half eye on this chat
but i can’t guarantee that i’m seeing it
all the time so
it may take some time to to read the
question and keep an answer otherwise
we will do it in the end so don’t worry
about this one
and well everybody knows jackhawk and if
you’re not j frog you know that jeff
rock has really cool t-shirts
so you can win one and uh i think this
time
perfect for corporate time if if you
have nothing to do to um at home or some
kids you want to
want to play with we have the traveler
and have to this nintendo switch so just
go to drivetrack.com
show notes and here you see the link
just go there i will share this later
again
so don’t worry about this one so the
topic
tests it’s an easy word short
but it’s quite tricky if you’re looking
at it so what
what is part of a test so if if you’re
looking at tests itself
and to understand what a test engine
should do or what a test engine have to
manage
it’s mostly you have this blue thing
this subject to test so if you have
whatever you want to test so i i don’t
care right now
if it is a cool java test it’s a web
test if whatever test
you have some preparations before the
test because you you need some
state then you want to run the test and
then you have some clean up phase it
means
you have to do all the work to to make a
clean environment again
so that you can start with the next test
and then you’re preparing the next stuff
so that means that for every test you
have this preparation and this clean up
and this must be a very
safe very robust operation so having
mistakes or bikes or whatever they are
this would be a
beast to find but if you have a bunch of
tests you have some kind of global
preparation so
what what could it be so if if you’re
testing for example
and you have a state you’re just ramping
up data structure putting data in
uh you have it you’re running a test
you’re destroying this data structure
that’s
it but if you have some kind of web test
it is
more complicated because you want to
ramp up resources you want to
start a surface container database
whatever
it’s all the stuff you so you have some
global preparations
here i’m showing just two layers but
have in mind that we have multiple
layers so it is
it’s just an antler thing so you can
cascade it in a way
that you have a quite complex structure
and um yeah this this must be
manageable so in a way that is reputable
so if you want to run tests twice or two
times
you want to have exactly the same result
and this is not trivial so there are a
few things that’s
that are not so easy to handle even if
you have just
easy piece of source code you want to
test so
but we’ll record this a little bit later
so what what are the bad things here
what what is the tricky part here
if you’re looking at this one it
implicitly means that you have
some kind of context so if you have a
preparation
and you’re creating a data structure the
first thing
is that you need to know who is holding
this instance this is instance holds for
the instance it
holds all the tests this is something
that is hold by a contact that is
outside of the text
it’s around the test so what who’s
holding who’s managing those resources
right where
it is and if you have this context have
in mind
that you can have there’s some bugs so
even inside this context of this
is part of the engine that is managing
the context
and with this um you have to think if if
you want to go to
concurrent tests how to isolate every
single context against
each other so with all complex things
you
want to have in mind so there is no
limitation because with job you can do
everything you can destroy it with
offline stuff or whatever so it’s really
up to you what you’re doing but your
test engine must be able to channel this
context in a in a performant way
because if you are looking at this you
see that
the most stuff that you see here is just
burning money the only thing that has a
value
in terms of money is that that you want
to
earn somehow is the subject to test
this is the only part that is earning
money all the other parts are just
earning money
that means you have to make sure that’s
ramped up and run down it’s
as fast as possible but the maintenance
of this stuff is just
as easy as possible so that’s this is
maintainable
over a long time and that
this is not leading to more challenges
than the test itself
so and this you have again cascaded so
you have it
for test you have a test suite you have
it a global preparation
whatever and um
here i i just
i just say that i assume that everybody
at this point
or mostly it’s a symmetric thing so
you have a ramp up and a corresponding
ramp down but this is not always the
case so there are situations where you
have an asynchronous behavior
and this will be tricky as long as you
have this symmetric
way of one went up one run down that’s
corresponding to each other
then the plane utilities you have or the
the basic
stuff that you have with jm5 is just
good perfect so easy to enter how to
deal with exceptions
with exceptions it’s if it is an
exception inside
the test itself it’s not not the
challenge but what
is happening if you have exceptions
during the
preparations phase this is some kind of
easy
because if the preparation is failing
the whole test is failing that’s easy
so everything else is good but what
happened if it’s
failing during the cleanup phase so in a
way that you’re not
able to see it immediately or inside the
context or in a leaky memory or whatever
so this is not easy to to analyze and
that’s not easy to
uh to deal with even if you have long
running tests
and you have this uh very slow
increasing memory usage or whatever so
just have in mind that you have an eye
on this one
and one thing is if you have different
tests you need a bunch of different
preparations so
make sure that or what what is a
the trade-off between making it generic
and making it specialized so this this
is the basic ideas
if you have now your with test groups or
a bunch of tests for the global
preparation you will have a bunch of
test groups so for every test group you
can scale so you can have
as much as you want test groups
whatever the name is tesui the test
group whatever
i don’t care about it you have just a
bunch of groups and these groups have
global progressions
but the really cool thing with jmf 5 is
that you have even a bunch of test
engines so and that means you can
have in one run multiple test engines
that are running one after another i’m
not talking about
in which order they are running i’m not
talking right now about how to activate
deactivate them but
in the end you can have from the idea a
bunch of test engines
that are running one after another so
that means if you’re designing your test
engine you have to think about
what is going on before your test engine
instance is running and what’s running
and what’s happening
after your test engine instance is
running so
um yeah well i
i think this uh dealing with states is
uh a typical thing in java
so what we have so far so this picture
i grabbed from the original
documentation i think so thanks for this
one to the j15
and the first time i saw this this
picture
i thought yeah it’s easy so i i just
jumped over it but then a little bit
later i thought
no there there are some tricky things in
so for example the first thing is you
have this old test you have this vintage
engine on top of the platform so
this is good so you have a platform the
fundamental
place for all your test engines and you
will get at least
two test engines with your installation
of gmod5
so if you are using it you have the
vintage one you have the jupiter one
the vintage one is for the old
yeah you will see what’s going on if you
have tricky beasts running with with
rules and all this stuff and all this
legacy
some some things are running some things
are running well
and some parts well you have to work on
but the good thing is if you can run
multiple test
engines you can think about
migration about slow migration this is
good because if if you’re creating
at one time your own test engine you
have to make sure
that you can easily migrate to a another
implementation or
back to the generic one or whatever so
migration is always a topic
even even during testing testing is our
production code
and with this production code we are
producing the production code for the
customer so
uh we should create test code like
production code for the customer
so it’s just our production code and
that means i have to think and calculate
in the same way so
i have to think about how to maintain it
how to get rid of it how to change it
and so on
so um talking about the old stuff and
the platform
then you have the engine this is a new
fancy
j105 environment everything is green and
shiny right now
and everybody’s happy to work with this
one and then
we have some kinds of use cases where we
need or want to have
third-party test engines we will go
through this one later
but um if i’m checking this one i
i thought yeah it’s clear but then this
bubble
at the bottom this is this is something
that yeah that was stealing some lights
so you see that ides and build tools
eclipse maven
and so on they’re all running the
platform
and if you have in mind that something
is invoking the platform
and just thinking about oh there were
they are
dealing with reflections somehow in
class parts and loading and all this
stuff
it turned out that in some situations
it is a difference if you run your stuff
in eclipse or in intellij so your test
plan is slightly different
or between maven and gradle so it
depends on
class parts on loading on how this stuff
is done
so if if something really weird is
happening
just try if you have the same results if
you’re just switching between
the ide or whatever just just as an idea
so and if you’re building your own test
engine you have to deal with some
transporting
class loading stuff later have in mind
that your test engine will be invoked by
a bunch of different tools maven gradle
eclipse
intellij macbeans whatever
and make sure that everything that you
are doing inside your test engine
is as much as possible proposed against
this
tiny little changes in this class past
universe
and then i’m not talking about models
and all that stuff this is just
just more so
j0.5 so we we have the subject to test
as preparations
and we have the global preparations and
all this stuff so we have three
parts with j5 that is
defining more or less one test it is
a test definition itself here it’s just
a test
then we will discover a little bit way
how to extend this one
how to customize this one and then we
have the extension context itself
so this three parts they are the main
parts of this life cycle and the life
cycle management so what’s going on the
test and how long it takes
and so on and this we have to understand
and we have to
learn what’s going on in the engine
because this is the test engine we need
later
which is our test engine throughout this
checkmark challenger
i think it’s very familiar that test is
just
inside the class a method with the
annotation at the test
so it looks trivial right now but there
are a lot of stuff we can customize if
we want to
create our own test later so here i
assume
that the test definition is always
inside
a class so a test is written as java
code
which is it’s not limited no limitation
so you can define for your test engine
test in a way
whatever you want it mustn’t be sounds
great just have reminded me creative so
you can do a lot of stuff so it mustn’t
be source code here i’m assuming that
it’s just
source code and in source word i have
this notation add test well that’s it
and uh we have some difference between
trading and four and five so in terms of
um must it be a public class or not it
could be it mustn’t be a public
class and it must be a public method so
but
this is just untactively chromium stuff
here you
can just read it in the documentation so
i don’t want to go up here in detail
but i want to customize it because if i
want to write
a test or if i want to create a state
and clean up the state i need some
possibility
to extend the lifecycle to customize the
lifecycle
and the nice thing here in general five
is that they
handled it was more or less
like like events you have the before
each event after each event you have
before all event and so on and every
event
is um represented by a functional
interface
so here you see i i want to create an
extension
that is manipulating the state before
and after the test
and then i have to implement two
functional interfaces simply for
each callback and the other integral
there is some
law of this lifecycle events just go to
the documentation
it’s a bunch of extensions or callbacks
and then you can see what’s the order
they are
um they are thrown or invoked
and so on but in the end you just have
a regular class here i’m i’m just using
it as inaccurate which is why
this extension done static so but
um you have this before each callback
and for this you have to implement
it before each method and then you have
the after each callback
and to activate your extension even if
you have
nothing in this extension it’s just
using this
extend with annotation class level or on
the method level
and then connect the class so
my extension was the class you want to
have associated with so
here it stands with my extension at the
class i want to have it so that’s the
easiest and straightforward thing to
extend now you have the possibility to
do something in before each and you have
something
the possibility to do something after
each so it means
you will get this instant of extension
context we will have a look at this a
little bit later
then you can do some code then the test
will be invoked
and then you have uh can do something
that um
is done after the test so far so good
before we are dabbing a little bit into
how to deal with this extension context
and all this stuff i just want to see
how i can deal with multiple extensions
so this you need to know because if you
have extension a and extension b
there’s one interesting thing how to
activate all of them
how to combine them and what’s your
order they are invoked
so here if if you are doing it straight
then you can just annotate
your test class with all extensions you
want to have
and the next question is in which order
they are invoked and this is different
so if you’re writing
extend with a and then extend with b
it’s different to
extend with b and then examples a so
water matters
so this is what you should have in mind
and latest at this point
it is clear that this way of extending a
bunch of or
to extended flowers with a bunch of
extensions in this way it’s just not
refactoring savings it’s just
just a question of time that your
colleague will do it slightly different
it will be a disaster so it will just go
wrong
what’s done so far if you write it in
this order first b
then a it’s called from top
to bottom so it will start with before
all
b then it will be called called the
method from before all
a then you will have before each way
before each a
then the test then after each a after hp
after all a after all b and this is
good so they’re going one two
test to one bag so this is very
symmetric and it makes it very easy to
handle because
if you have the rampable database first
and then the software container
it will be invoked in a different way so
just destroy the circular container and
then
destroys the database so this is easy
and straightforward and for the most
uh things this is just the right way to
go so it was a good decision to do it
like this
but sometimes it doesn’t fit that’s it
but have it in mind
so if you want to maybe
it’s a good thing to just give a um
a quick motivation uh for this
it’s a good way if you want to extract
it before like a method that you wrote
before a before test and after test
something that you know from junit4
and you wanted to to apply it for let’s
say 10 different tests but the same
before
before test and after test so this is
coming
so you can extract them so just people
to have the motivation
and why why would they even create that
yeah okay this is more or less what’s
coming up next so how to how to maintain
this stuff
so um but but the main thing is if if
you start dealing
with with life cycles or with
preparations
um some some developer um
want to do it over inheritance
traditional way so so i have my base
test
and then i’m extending my base test and
then i’m going on the inheritance and
this inheritance means that you can just
inherit from one
parent and so you have a linear list of
inheritance
and the other thing could be that you
have a bunch of interfaces with
stateless stuff
and then you have a composite of these
interfaces but both
ways are not really what you want to
have
so too tool um if if you have a bigger
project and you want to make sure that
the surface container for example for
this project is invoked always in
some yeah in some way that you need it
for for this uh project
so you you would have one developer
that’s creating this extension
i’m ramping up our surface container so
and here he assumed that the database
already there and he assumed that after
his preparation has done something
whatever he is just focusing on
how to ramp up the traffic container
then what you can do you can
write this extension you can make a jar
out of it
give it a version push it to your
artifactory
and use it somewhere so it’s just a
regular piece of code and you extracted
some kind of
functionality so it’s just a
decomposition of
a big bunch of functions two
tiny pieces atomic pieces so that you
can compose them
through more complex um pieces of code
and if you want to compose it
it’s more or less you have this atomic
pieces and it’s really good advice to
think
in an extension that is more or less
atomic so that you don’t
think about what is coming before what’s
coming after so it must be
in the ideal world something independent
and this is basic extensions
mostly have this technical names like
ramping up database ramping up
software container whatever and the next
level would be
that you build a composition for example
i have a
web test a web test assumes in my
project
first database then servlet container
and that means
if you have a composite of extensions
you start now with this those extensions
so this
that you annotate your um interface
this annotation here with add extensions
then you are adding all your extensions
you wrote in the
order you want to have them invoked and
then you have
one annotation you can use for your test
so it means the wording in this
piece of code is more domain specific
and so basic extensions more technical
so
with every layer you’re adding you will
have a more domain specific
wording and then you have later for the
developer just
add web test at database test at
whatever test
or at customer whatever test so you can
start
building a complex hierarchy based on
automatic thesis of extensions
and the next thing is if if you know how
to
compose this hierarchy the next thing is
who’s
holding the instance you’re creating so
for example if i’m creating
an instance of my undertone or tomcat
restoring this instance because if i
have
different if i have different tests
running concurrently i need
concurrent invocations of topcats of
different
tomcats i can create them with different
ports or with test containers whatever
and then who’s holding this instance um
it’s it’s not recommended it’s highly
not recommended to to have an attribute
in your test class that is holding this
instance
because it’s not sure that you are using
always the same instance of this test
clause or it’s
invoked several times or whatever so
don’t don’t trust this one
but there is this extension context
since the extension context small is a
hash map
so you have this context and then you
can grab out stores
stores is just a naming schema and this
name
will give you a key value store and in
this key value store
you can put um the instance here i’m
just putting a
string in but you can put in the
software containers and so
you undertook and then storing it and
inside the test you don’t have access to
this extension context
but just during this um callback phases
so if you have before each and after
each to get the same
instance you’re just going to the same
store
with the same name here i’m just using
the global one it’s not perfect but
so you have to think about a scalable
naming schema
then you can say okay give me from the
extension context the store i want to
have
and here is my instance i i want to deal
with
so and then you can run down during the
test you don’t have access to this
instance but you can use for example
for instance that is running so
um the next thing what you need
sometimes
is the possibility to inject something
into your test
because the extension context you don’t
have access to this instance
but if you want to have an in it during
the lifecycle you want to create
something and then you
need this instance inside your test you
have this injection mechanism
and for this you need this
parameter solver a parameter resolve is
just an interface again and this has two
methods
supports paren here what you’re doing is
just you
define how to identify that this
instance that should be injected
is managed by this parameter solver
for example here i’m just checking for
an instance markers
interface or whatever and then the
method resolve parallel
will be invoked after sports parameter
and then you will get the same
informations in parameter context or
some context from the invocation itself
and the extension context
and then you can correct the information
you need out of it and create an
instance of whatever you want to have
injected
so this is a basic thing but um
well for the details you need to read
the documentation for sure and there are
slightly a few difference between
the you know versions of jln5 so i have
make sure that you’re reading
exactly the right documentation for your
uh version you are using
but this is more or less everything so
you’re identifying the type you
have the decision i’m responsible for
and then you’re creating this stuff
and here you can do everything there’s
no limitation you can ask your database
you can do whatever
whatever magic in java is possible you
can do here
to activate this parameters over for
example
for method you can just extend with and
then declaring the class
so what you have right now is you can
manage the life cycle with extensions
now
and you can get stuff injected from
outside
so with this you have a lot of tooling
that will help you to
test later your test engine or whatever
your regular test
so
it means if if i want to write a custom
test engine i’m writing a piece of code
and this
means if i’m doing tdd then
i want to write a test from a test
entrance so what i need for this one
i need a way to test the new test engine
and for this it’s highly recommended to
use the jupyter engine
because this is the best out of the box
test engine you get
so use this one to test your test engine
and if your test engine is at some
development stage that’s you say okay
now i can test myself
feel free to change but um the first
recommendation is just
go with this one what i want to have
uh i want to first um see what what is
the test import so for for this
test engine the input is source code is
a test class
and i have here an example that is one
positive test that has
one the test two is just um
throwing a runtime exception so it’s a
failed one and the third one
is disable test and now i want to check
how i can
test if the test engine is doing exactly
what i’m expecting
so for this you need a dependency in
test scopes uh
to jump platform test kit with the
version you want to have or you are
working with
and again here i have one positive test
one test so it’s failing and one that
is just disabled and how to identify
that the test engine is doing exactly
what i’m expecting so how to test my
test engine
i have this class engine test kit
and there’s this static method engine
and here we will
cover this later what an engine id is
but i will give you the engine id from
the test engine i want to
test here it’s junit jupiter
then i define what is the input for this
test engine
and here i’m using the selectors i have
done static imports so if you’re dealing
with reflection just
check what what support utilities is
part of this 2 unit
um depends who you have already because
they have done
really good job with reflections so
don’t write this stuff by yourself
try try this one so and here what i’m
doing is i’m selecting a class
and i’m selecting exactly this class i
defined before
and that means this is my test input
so and then i’m executing the whole
stuff
and i’m grabbing out the tests and then
i start analyzing it with the search
statistics and this means
i check how many started how many
succeeded how many skipped and how many
failed
and then you see okay two tests started
one is disabled one was positive one was
kept and one was failed so
exactly what i’m expecting and this is a
way how you should test your test engine
later
too so you have to define how to
define a test and what you’re expecting
and
with this you start writing tests
against your own test
engine to make sure that the behavior is
exactly what you
want to have so uh
this one
yeah so yeah okay
so now i want to test what’s what i will
get out if i want to have the dedicated
behavior or there’s a specific behavior
of a test so if i’m just checking if the
taste is good
or bad it makes no sense so i need more
information for example i want to
analyze
if this test two that’s failing is
exactly failing in a way
i’m expecting it and here i’m doing
exactly the same so i have my test
engine
i’m selecting now the method of this
class with the name test002
this is not nice because it’s not
refactoring safe so this string
is not drawn to the method instance
itself so that’s a weak point here but
you’re defining one method so now i
haven’t just input just one method
and then i’m executing this one the
first thing is i’m just checking okay
this one failed
exactly what i thought and then i can
grab out the
events and the characteristic this is
detailed information
so it’s finished with culture yes it’s
um it wasn’t runtime exception yes and
there was a message
exactly with this one so you can check
what kind of exception happened and if
it is exactly the
input or the the message of this
exception with this you can test against
villages
i have this one so i can now test i can
invoke a test engine
i can uh define what’s the input of this
test engine for this run and then i can
analyze
the results after a test engine run was
successful
so the next thing is how to get this
test so a test
engine should discover somehow the test
then i prepare the launcher i’m creating
a test plan
which is what actually should be invoked
in
in some way and then i’m registering
some some kind of listener i’d say
they’re just collecting information
during the run and then i’m executing
everything and consuming
the result so i’m not expecting that
everybody is reading the whole source
good and and remembering it so you have
some slides uh go to
drivewrap.com show notes or check on
the internet so if the slides are
available otherwise ask me
and then you can go to this in detail
what i want to show you is a
general behavior of the general way to
go
so what you are creating first is a
request so you’re selecting the input
you want to have for this test for
example here i’m selecting some package
so what’s this request um this one i
need
and then i need instance of listeners
listeners that are
collecting information during the test
run here i’m just using the summary
generating listener the default one
that’s
part of this and the bad thing here is
that
uh so far i know you have to hold this
instance because if you’re creating this
instance and um using it during the
creation of this configuration then
you’re not getting
the stuff out after after the test run
is uh
done so you have to hold this instance
of the listeners
then you’re building the configuration
it’s um
i highly recommend that you’re switching
off all this auto registration and
whatever stuff so that you have control
what is activated and what’s deactivated
then you’re creating one instance of
your test engine and you see
the jupiter test engine is just new
constructors just
creating instances so easy so no magic
and then you’re combining those in the
configuration and after this
you’re just running it uh it means first
you
are discovering the discovering is this
reflection magic that will build
the test plan if you’re defining your
test source good
but feel free to build your test plan in
any way
you want to hack so it could be
something completely different
the only thing is you need some instance
of a test plan that’s it
then you’re executing the stuff and
after this you’re grabbing
the information out of your listeners
from the test run
all this is done you will have what’s
next
so the basic structure to create a test
engine
the absolute minimum is i don’t want to
have a real test discovery i just want
to have
test engines that will be invoked and
i just want to see if the test engine
will be invoked by navy by the ide and
so on this is the first uh
useless test engine what i need so far
is i need in compile scopes of juniper
platform engine
and then it’s a platform comments in
compulsgate
scope yeah and then
yeah sure i’m not sure that i understand
why do we need test engines at all
test engines okay so why why why should
i write my own test engine okay
um the main thing is if you have
if you have complex lifecycle for
example and you want to
um manage this one for example if if you
have um
every infrastructure that you want to
run before the whole test plan is
running so far i know you have to do it
outside then you’re running the whole
test plan and then you can ramp it up so
your
your life cycle is not managed by junit
completely anymore
and or you want to have something like a
license check for example
because you’re building a product or you
want to have something like flaky tests
or you want to have something like
um uh what’s this called this
um rotation test cover uh for example
mutation test
engine or you want to have this other
test techniques or you want to create
your own dsl
you want to have something like a
behavior driven or whatever
domain specific language uh to give the
possibility to create
or to define tests so all this stuff
you should start thinking about creating
a test engine
because the generic jupiter engine is
good for the most cases
but if you’re going to specific cases
the the way to create all the stuff for
example if you’re running
web tests you have a complex life cycle
you want to ramp up selenium nodes
you’re dealing with stalker images and
all this stuff
and sometimes it’s easier if you’re
writing your own test engine for example
in this case
and makes a whole lifecycle management
easy
or if you want to integrate in existing
infrastructure
you need to connect to other resources
or you want to distribute tests for
example i wrote a proof of concept for
distributed test engine so that you’re
collecting your stuff you’re pushing all
this into a hazelcraft cluster executing
it concurrently and then collecting the
information so
writing this one with a regular journal
for jupiter engines just pane
and then writing your own test engine is
straightforward
and sometimes yeah you have to decide at
what
complexity level you’re switching to
your own test engine implementation
i hope um this was a short
short while um
any other question to this one right now
okay so um to write a test engine is not
not so complex
and sometimes really it’s easier as
writing a complex life cycle with
extensions
um the only thing you have to do you
have to implement the test engine and
then you have exactly three methods to
implement so
get an id this unique id you need to
identify this test engine
you need to discover your tests here
feel free to decide
how to discover tesla if you want to
consume a database to grab
informations out of it to build a test
plan against whatever
if you want to read text files to create
your test plan if you want to
whatever you are thinking about here is
the right place to
to identify how to create a test plan
and then there’s this execution phase
just
you have your test plan and now you’re
going recursively to all childs
in this tree inside your test
descriptor or your test tree and
executing this one
so this is small is the basic what you
need
and um yeah that’s that’s
let’s go to to this one if if you have
done exactly this one so for example
here i’m just
um giving you an idea whatever you need
i just me
taking the simple name then i’m creating
exactly one node so root node
for my test plane with no test in it so
i’ll just write down it
and then what i’m doing in the execution
phase is just for my request i’m
grabbing the root
the test descriptor from my my start
point
and then i’m grabbing all listeners out
of it i’m
executing this stuff so i say okay now i
start executing my test
i’m going to the engine grabbing all
children out of it
going to every children saying to my
listener whatever listener you want to
have here just have this execution
listener
say okay i start executing i’m finishing
successful or failed it depends what
what is the result of it here i’m just
saying successful
and after this i’m just saying okay i’m
done so we’re going recursively this
tree down
and collecting the information about
every single test he has no tests so we
have just a
success going to the root node and
that’s it
so with this yeah yeah
sure um maybe i’m misunderstanding where
is the test actually run
the test is running we will come to this
with the next
next um test engine this is just a whole
thing but
during the execution during during the
time the execute method
will be invoked you’re going to all test
nodes it’s a tree
and then you have different kinds you
have container tree nodes and you have
test tree nodes and if you are
going to the test node or if you have a
test node
you will have all the information you
collected during the discovery process
for example class what method what
whatever date time
whatever you need and then you have the
possibility to say
execute now and this is the place where
you’re
actually really executing the test
itself
okay yeah we will come exactly to this
one so
um so if if you’re just going off the
root note with no test inside
you will get already the whole support
that you see i have one container
started one was successful so
it’s it’s running so you can use maven
already if
you activate it i will show it how to
activate it but this is
just a valid test entry it’s not testing
it’s completely useless but
so how to define your own tests so for
example that you can vote i have my
manual test my vardy test my databases
whatever
for your own test engine so we have to
create an annotation
then we are executing all uh we want to
discover all tests we annotated with
this normal test
and then we execute it and what i want
to have is next is that the ide is
supporting it and maven is supporting it
and then you have more or less
everything you need at this point
to build this marker you have just to
create an annotation with
target on a method for example if you
want to have it or method if you want to
have it a method in
class then you’re just using element
type method and
clause this is good if you want to make
sure that some annotations with
information’s insights annotation is
just used on a class level because you
want to have this one
for init processes and some others just
on a test method level
so that you’re not mixing up attributes
for example for global preparation and
for local preparation or whatever
the only thing is that you’d really need
is a retention policy runtime
well otherwise you can’t grab this stuff
out with reflecting later
and there’s one thing called testable
and this is the only thing you really
um have to understand why this is there
we call this testable so here i’m doing
this one here for example for the
target on class i want to have one
iteration that is just
for initial class level and this one was
just for init on
method level if you want to divide it
it’s up to you but the only thing that
is a little bit magic this is testation
and testable is more or less they um
i think it was coming from jet brain so
they they needn’t the ide
needs a possibility to identify what
annotation
is actually a test annotation to support
it
and you’re completely free your there is
no inheritance or whatever
so you need some kind of identification
and then they defined
you have this this annotation test able
and the ide vendors are listening or
checking for testable
and if they found this interface on this
annotation
they will treat this interface like a
test language unit and
test annotation but then for your
present
so that is why you need this a test
table so that your id is supporting it
and after this it’s all ready so
to have this runner engine just a new id
because we have a new engine
and the main thing is you have to define
what is a test class
or test method uh for this
here’s a part where you can define the
syntactical behavior of it
so how to define the test so you can say
i’m not supporting extra classes i’m not
supporting
private classes and i’m not and i want
to have
explicit annotation nano test class so
then i will check if
i have for example some some other
attributes so here you’re defining what
is a minimum requirement so that it will
be treated as a test class
so i extracted here this one just as a
predicate
because we need it later so everything
you need to define
plus the annotation that is um
the marker for your test class if you
have this for test methods
for example you can say okay i’m just
executing it is
not static if it is not private if it is
not abstract
if i have a param count that is exactly
zero
or you can say okay i want to have at
least the param
of type a or of type b so here
you are defining the syntactical stuff
how to define a test and this is more or
less
the the first step to define your
requirements
for the test environment if you want to
make sure that every test method will
have an inject at
whatever test class or report or
whatever
here’s the right place to make sure it
is just grabbing this one
then annotated with number test okay
it’s a test method
and you can define if you want to have
some kind of return type
for example if you want to have
statistics or some kind of return
from every test method because you just
want to have it for whatever
then you can define here okay a test
method must have a return type
a or b or whatever so here you can
define everything you need again
just a predicate we need it later during
the discovery process
the discovery will give you four entry
levels and this is a little bit tricky
if you’re doing it the first time you’re
you’re just going here a little bit mad
and i’m not sure if it is already
changed but you have to
write copy paste this boilerplate code
by yourself it’s
it’s just pain i hope they will change
it but it’s always the same so
what you’re doing is you have this
classplus root selector this package
selectors is class select and method
selector
and here’s the change so if you’re
running with maven for example on a
package
then you will get a package selector if
you’re using exactly the same with right
click on
intellij you will get a class plus root
selector
with a filter so you have to implement
all these
four ways to uh
to work or to deal with the selectors
otherwise
the behavior is slightly different or
some something is just not working
but in the end what’s what’s going on
here is just
you’re going recursively from this part
here
you’re in through all entities you have
you’re cropping every single class
going in every single class to every
single method and checking with your
predicates if it is a test class and if
it is a test class if this is a test
method you want to support
so this is what’s going on here um which
is a bunch of code it’s just going
recursively down so
okay if you have a class check if it is
a test class
and then append
all methods toward the next test level
and so on so and if you have
uh found something we’re creating a
descriptor
and the descriptor you will see the
implementation
it’s more or less uh right let’s see
here the descriptor
is the node inside the test plan and
this is a place where you hold all the
information
that you need during the run time during
the test execution
so you’re collecting here in this
container nodes for this
root node for example the class name the
method name
and some whatever input
so these are the descriptors
and then you are going to checking class
going over it creating this descriptor
and so on
so the main thing is you’re going
recursively
through all class passes we have just to
a package
just for class or just checking that
method
using the reflector you tilt from the
jail to 15 because this is making this
work
really easy and then what you want to
have is more or less
this test plan make a tree out of it
please no graphs
and then you have two nodes you have a
type container and type test
why you can have a test in a container
at the same time i don’t have that’s
possible
mostly easy it’s just hold everything in
a container or a root a node
should be always a container until there
are child
children and the children the last one
should be a test it’s the easiest way to
to define it and then have a look at
this abstract test descriptor
and don’t implement the interface
because there’s a bunch of work already
done
so you can go there as well so
how to deal with this what you can do
not much but you can what you can do if
you’re on a class level and
classes are the definition to define
tests
if you have a class test descriptor what
you can do you can
go from here recursively down for
example
finding all methods in this test class
checking if it is a valid test method
then just creating the next test
descriptor and adding this one so it’s
just a recursive way
so make sure that you’re giving back the
type
it’s the last line here type container
type test
so that later during the execution phase
it could be
identified right
so um coming now to to the information
you have during the execution phase so
the last thing uh the children for
example the last children in the street
um is the last night we’ll have all this
information here i’m just collecting the
test class and the test method that
would
that is identified if you have inside
your annotation some
some information like an attribute
inside your annotation it would be
good during the discovery process to
extract this value and store it
in this test descriptor as well because
this one will be used
during the execution a little bit later
so here is the right place to store all
the information you need
for the test that is static and not
requested
during the execution the executor itself
so now if if you want to execute it so
it’s just
it depends on the level you are if
you’re on the root note the engine
descriptor just say
okay execute this container start with
recursive going down
the next one we have class descriptors
if it is a class descriptor just say
execute this container it’s nothing else
that’s going one step down
and then the last one if it is a test
method test descriptor
we’re actually executing the stuff means
in containers just i will say to my
listener
i’m getting out of this request and say
i will start working on this level
then i’m going to all children and
execute them
and if nothing is happening i will say
i’m execute i’m finished i’m done and
i’m positive or negative
so this you are doing for every layer if
you have three layers in your tree
we’re doing this for two layers and the
last one will be
the execution method and here again i’m
now on method level
i start execution i’m executing the test
method and i’m giving back the results
the test execution result
so after we have done all this
repetitive work
really copy paste your stuff it’s really
don’t write it by itself
then we’re coming to the final execution
of the test
and that means i have this method test
descriptor whatever your name of the
test descriptor is
everything inside this descriptor is now
usable for the execution
plus everything you can request now and
then
what you are what i’m doing here is i’m
just using the reflecting video to
create
this test class i’m invoking
by reflection this single method and if
no
exception is happening i assume it’s
right and giving back
successful if there’s any kind of
exception i’m just saying
it’s failed that’s it if you need
technology here in this one like
cdi like whatever it’s the right way
here to get all this technical
infrastructure to execute your test
but this is the main thing so here you
can do everything that java is learning
and just do whatever you want the main
thing is something is happening
it’s failed if everything’s right the
feedback is successful
that’s it so now you are able to write
the whole
engine in a little bit more compact way
so getting an id
you have the two ways to define if it is
a test class or a test method
here with this predicates then you’re
going down
building up the test plan during the
discovery process
and the execution is just going over
this test plan
and executing every single test inside
this hierarchy
that’s it and you’re collecting
information when does it mean that the
platform is very very
thin layer uh yeah
it’s just a little bit called java so
the platform itself is just
just a few methods it depends so it
really depends so how to create
so one one tricky thing is how to create
your test plan if you’re just going over
source code
you’re just scanning your classpaths
identifying the methods collecting them
from
that’s it if you have a test engine that
is combining different resources for
example
at this date on this machine with this
class path
and whatever this configuration this
creation of a task plan is more or less
work and the next work
that should be done is execution itself
there’s just a bunch of software that
will be executed so
it’s a test so most people just thinking
that
uh a test plan should be just defined in
in source code but it’s not
it’s not a limitation so be creative
here because then you have the full
potential of this test engine
so that you have the wrapper to execute
all the stuff
the next thing is how to activate this
one
it’s just uh service locator here or
unit platform engine test engine
and then you’re just writing the class
name in it
and what you achieve now is you can now
take a class using edna test class
on the class level and nano tests on the
test level
and you have full ide support now you
have full maven and gradle support noise
so it’s it’s done so you have now a
complete independent test engine
that you can just wrap in a jar making a
version out of it
and let your team use it
why why or what can you do with this
test classes
so for example if you have a commercial
product and you want to have something
like license check you can do it inside
your test engine
on the other side if you have license
checks because you are consuming
expensive resources then you can do it
inside your test engine this
authentication
authorization process can be done inside
this test engine
um ramping up some specific
infrastructure
checking against uh limitations
something like a
failed break or some whatever whatever
you want to have you can
add in this test engine that is helping
you to manage the life cycle
and to make the usage of the test
itself easy the main specific test
definition
so if you want to have something like
behavior driven tesla whatever
some other way of defining test it’s
up to you to define the syntactical
environment you want to use
to define a test plan you can use
this um test engine for dynamic test
input of external resources for example
if you are not able to write it in code
java because you have to ask during the
time of the execution some external
system to get some
stuff out a similar test engine would be
a good place
uh compared to extensions if you have
different tech
testing techniques like mutation testing
or
property based testing or random monkey
testing
or whatever statistic you want to use
for your testing or
security payload testing problem all the
stuff you you can use
and dedicate a test engine for it that
is specialized for this
so that you don’t have to build with
generic tooling
this behavior this is mostly easier one
thing is you can create persistent
immutable test plans
if you’re creating test plans and you’re
doing it twice
and you’re doing it once was maven once
was uh intelligent
it could be that the test plan is some
kind of
different it depends on how to create
the set plan and it depends
what are the the weak points here like
reflection or ordering or sorting or
whatever
if you want to have immutable test plans
because you want to have this
test plan stored with build information
so that
everything is reproducible even the test
line here’s a good place to go
because you have the test plan and you
can store it in a way
that you can share it rerun it exactly
in the same way
what you can have for example is like
something like test execution priorities
based on the usage on external resources
or your own resources you can prioritize
tests on different machines on different
nodes whatever
because you can interact with
distributed system for example if you’re
spreading it over hazel cars cluster or
if you are going to whatever system
so you can do a lot of stuff that is
complicated with regular extensions
and complex lifecycle be sure if you
have a complex life cycle have a look at
this one
for example this i mentioned already so
if you have for example
the need for for distributed uh test
execution
then uh this could be one thing for test
engines just grabbing the classes
pushing it to a distributed class loader
to distribute
uh distributed execution nodes run the
stuff there
grabbing it back i have done it for
example with uh testing
um vardy nabs because vardin itself and
the web tests
with ramping up selenium and stuff and
databasing it was low
so i just scaled horizontally because
the chest itself is slow so i’m scaling
horizontally and
this was easy to always for example yes
because so
inside your test engine um
yeah then security payloading is one
thing you should have in mind for
example
if you want to have security payloads
for for your um
apis or whatever then you could do it
like uh you have a repository you’re
asking for the binaries
for the immutable you’re creating this
pass plan you’re injecting this stuff
and the test result will be distributed
for example in external reporting
systems
or on whatever so it’s
really if you have those complex life
cycles and need for documentation all
the stuff
and custom test engineers is pure gold
for it
because you can customize for example
auditing systems and all this stuff or
um creating a runtime metrics uh to to
see how much time you have burning on
every single machine
and so this one okay well
if if you’re spawning new nodes with
your test engine with stalker or
uh loading dynamically stuff uh it’s
perfect if you’re using out factory with
z3 to scan on the flight because we have
this rest api
and during the execution you can
provision the whole environment so you
can say
i want to have for every single test
step a dedicated repository i’m using
if one test is failing so one step of my
test pipeline
is failing i’m holding all states until
this one and i’m just
re-running the latest step of my test
pipeline
so i’m not doing all this stuff again
and again so you’re killing rebuilds
with this
so this is one one interesting thing and
we have some license check and all this
stuff
um yeah well so more or less i’m
from the technical point of view on i’m
done with the test engines i hope i
could give some ideas what you can do
with this one
for sure you can try this for free check
the website there’s some some
interesting news
so just just as a heads up and
um well i’m open for questions now
if someone has some experience or
some questions or some ideas it would be
just the right right time to to ask
too far away well
the main thing is um
just right if you want to have this
code the boring part was the discovery
just let me know
i have this demo somewhere on github i
think already
so that you can just start with ramping
up your own test node
and then playing around with this stuff
um
yeah you can do it with every everything
that’s running more or less on the jvm
so with scrolling or whatever
um then can i have a small question
sure do you have some recommendation of
a migration strategy between
jet unit four and five
documentation well not really
documentation
i i just know that i i only know that
mostly
people have trouble with with his rules
so how to migrate this so everything
that is inside the life cycle
that is more or less a pain so
if this is complex and then for example
if you have to
to rebuild this life cycle then
sometimes
it’s really it’s a good thing to um to
have an
own test engine but then you can rebuild
this lifecycle
without doing this stuff or you can
check for example if if you want to have
some special support for this
but that’s the only thing i know so
migrating if if you’re not able to run
it with a vintage
engine inside your junit 5 platform
and you really have to migrate this
stuff
well you have to check how much code you
really want or if it is just running
this
vintage engine okay thank you
yeah okay
hello so anybody has a
maybe a general purpose question
on journey five it’s a maybe a good time
to ask
well the easiest thing is really the
documentation is not bad for the for the
course stuff so
just start writing your first test
um going in detail um
if if you start writing extensions don’t
do the mistake and
and uh hold instance by yourself really
go over the extension context do it
um so i swore too too often don’t go
over inheritance
don’t do it so i have to do it like make
sense i said that’s also a really
overall
composition of extensions
um otherwise it’s a very
very straightforward thing sometimes a
little bit tricky
if you need for example this parameter
server to create a
dynamic range of input values so they
have this uh
important cvs or whatever stuff but if
you really
want to read the config based on this
config and the environment variable for
example
oh i’m on a windows machine and now i
want to create this web driver and
another web driver
so stuff like this this is um it’s not
so handsome
with core extensions so yeah
so if if you’re on this level then then
think about how to manage this lifecycle
even even
then later it’s not symmetric for
example you have
what’s the right time to create a web
driver if you have a bunch of web
drivers what you are doing with web
drivers if one test is failing because
then the whole test
execution is just killed so all this
stuff so
then then think about some convenience
with your own test engine
it makes your life really easier
so it seems that the real meat here is
inside the
um the engine themselves so jmd4 as we
know it is
everything essentially is inside the
vintage
implementation of the test engine right
so the platform seems really really
thin yeah it’s just execution of a test
engine and this is
executing three methods more or less
that’s it
so idea discovery execute that’s it so
this is a
thin layer uh what’s really good
is this reflection stuff they say build
so i really recommend you
to have a look at this one because
there’s a lot of work done inside the
shoe reflection rituals
um but give it a try
so if you have your blueprint for test
engine it’s just
a template project and then defining
your new syntax because
the boring part is the discovery if
you’re just going over classes
then the discovery process is always the
same maybe they extracted it now for
with a
better extra class what you never should
do
my personal experience maybe i’m wrong
but what you never should do
based on my experience is extending the
jupiter
engine don’t do this one this just
no don’t do it so um
everything you are not able to do with
extension towards your own test engine
but don’t extend the
or inherit from from the jupiter engine
it’s too complex
i wouldn’t do it
there are the ones just tried it’s
really easy
sorry any more questions
oh quite i saw here some some some
questions on slack
some corrections from chris uh uh not
reflective utils it’s called reflection
support
sorry for this so whatever
reflection supported there use it from
the gl5 team
don’t write it by yourself that’s the
main message i want to say
anybody else has a question about unit 5
you want to ask
oh yes a simple way to migrate in place
from james to jm5
simple way of migrating the test itself
mostly mostly it really depends on
how complex you have done your chair
your neck force off so if this was
good and maintainable written not too
freaky in terms of inheritance and
mixing up stuff
it will be easy it’s it’s just a regular
prefecturing project so it’s it’s like
it’s your production code it’s it’s the
same as as you have done for your
customers so
um mostly the pain mostly is that this
source code base for 2004 project if you
have a legacy project it’s
so old that you have a huge amount of
layers inside this one
and to get rid of this you really have
to
to define if if you want to rewrite it
or if you want to migrate it so
um but what you should do
if so what i personally do is
if i want to migrate i’m just
creating not the line coverage i’m
creating the notation test coverage
because this is a way stronger coverage
and try to increase the existing test
environment
the mutation test coverage and then
migrating because this is a
there’s a stronger one compared to line
coverage i have a talk on on youtube
about mutation test coverage or i can
give it here again so
um feel free to check this one out
because mutation test coverage is really
it’s a strong thing um
the context of the maiden face because
in a mixed mode it doesn’t run in
parallel so much but in a negative force
for say just jam private runs okay
yeah so there is from from chris is a
one-on-one implementation so i’m not
using this surefire whatever plugin i’m
using this one
from uh accuracy so um
because this makes my life very easy so
but i can can
check the link to the github repository
and share it
so i’m not using file save and and show
fire anymore i’m just using this one
it’s tiny it’s fast ah here already
here’s the unit platform maven plugin
we’ve posted it in the chat already so
this i’m using introduction now and
it’s it’s solving a bunch of
challenges i had especially with show
fire
so um otherwise concurrent execution is
always a pain if you’re not clear with
states so it’s
like regular code and um
i think enjoy unit 4 a bunch of stuff
must be static and this is painful and
this is
just not existing in general 4×5
but um
as far as i understood it right there’s
no guarantee
how often an instance of a test class
will be created so if you have three
tests inside this test instance
it’s not guaranteed that it’s just
invoked once and then
uh invoking all three test methods or
it’s
in uh creating for every test invocation
anyways and so
there’s no guarantee it means you have
to make sure that every single test
and is perfectly just atomic
sorry and it means don’t don’t do
aesthetic in all that stuff so
it’s just regular if you have to deal
with concurrent
immutability concurrent uh stuff no no
static don’t don’t use barrier legend
whatever stuff
so
okay we have a question in the chat
for mark and you can see it and second
for blast
um i think i saw something
so i’m not really sure but what you
should do is this
pioneer project so everything that’s not
going into the core j155 project is
collected in this pioneer project
and um i think they have a list of
already known
test engine custom test engines so have
a look there
but i personally never have never used
it so i can’t say yes or no
i see create chris answers that octo
will be 100
running on the jrd platform okay cool
dude
oh okay so when if there are
no more question then i think it was
very
very knowledgeable as well
thanks a lot thanks a lot for sharing
this information
i think it’s sometimes a it seems a lot
of information but it gives the people a
spark spark ideas that sometimes you
know
get cooked in your brain and you use
them later
yeah so that sort of gives a really good
ideas so thanks a lot for sharing
and then and for the community we’re
going to see you we have a
next in september we have two lectures
and
also seem quite interesting so hope to
see you all
in september lectures