Article
Case Study: Capturing Quality Images for ML Algorithm
This post was previously on the Pathfinder Software site. Pathfinder Software changed its name to Orthogonal in 2016. Read more.
As sort-of promised in last week’s post, I’m going to work through a real-world test example, with an eye toward explaining how and why I tested the way I did. Hopefully, I’ll be able to do this at blog-post length. If not, well, there’s always next week.
This site, which was a legacy rescue, allows users to send messages to each other within the site without having to give away their other contact information. The problem is that nefarious spammer types were creating logins and immediately sending messages to large numbers of the user population, irritating them. After some deliberation, the client decided on a rate-limiting strategy, where a member could only send a certain number of messages in a day, and a new member could send even fewer messages a day. Messages above that point would require administrative action to unblock the user’s privileges.
I started with Cucumber tests — the first two are representative of the initial batch. Everything you see here is actual code, only slightly tweaked to anonymize details of the site. But I’m trying not to present simplified “example” code. That said, this is the final state of the code, there were some intermediate steps and false trails that I’m sparing you from having to read about. I’m also focusing on the tests here, rather than the resulting application code.
Background:
Given the user database is cleared
Given a group of recipients.
Scenario: Normal User behavior
Given I am a user who is not a new member
When I try to send 5 messages in a day
Then 5 messages are sent
And the administrator does not get an email
Scenario: Excessive User behavior
Given I am a user who is not a new member
When I try to send 6 messages in a day
Then 5 messages are sent
And I am blocked from sending further messages
And the administrator gets an email
This test uses an implicit style, so a lot of the details are in the step definitions. I start by writing the step definitions one by one until I get to one that I can’t make work without writing new code. As it happens, since not sending email is the current state of the system, the entire first scenario should work without any new code. It’s very important to include that scenario, though, to ensure that the new changes don’t break the basic behavior.
Here’s what the steps look like — the user who is not a new member defers to the boilerplate RESTful Authentication step for logging in:
Given /^I am a user who is not a new member$/ do
@user = Factory.create(:user, :created_at => 5.months.ago)
Given "I am logged in"
end
Given /^I am logged in$/ do
visit "/login"
fill_in("email", :with => @user.email)
fill_in("password", :with => @user.password)
click_button("Sign in")
end
Next up, sending messages. Since the point of Cucumber is to treat the application as a black box, the step definition uses Webrat to simulate number of posts to the create message RESTful action. (The @recipient
is created in the background action, which I didn’t show here because it’s not very interesting.)
When /^I try to send (.*) message(.*) in a day$/ do |count, plural|
count.to_i.times do
visit(messages_path, :post, {:recipient => @recipient.id,
:message => Factory.attributes_for(:message,
:sender => @user)})
end
end
So this step definition allows you to match “send 3 messages” and “send 1 message” by adding that little group at the end of the message, that group needs to have an associated variable in the block argument list, but it’s just ignored. Somebody with better offhand regular expression skills could easily make it so the end of the expression only matches “message” or “messages”, but I can live with it being a little overly matchy for now.
Closing out the normal case, the step definition to test emails sent uses the excellent
email_spec plugin, which creates RSpec matchers and Cucumber step definitions for email testing. This one checks an all_emails
method, filters out any methods sent to the administrator, and makes sure the right number exist.
Then /^(.*) message(.*) (is|are) sent$/ do |count, plural, verb|
emails_out = all_emails.select do |e|
!e.to.include?("admin@admin.com")
end
assert_equal(count.to_i, emails_out.size)
end
The administrator email step definition uses the email_spec steps directly — and yes, I could have included those steps explicitly in the Cucumber scenario. I chose not to, on the perhaps dubious grounds that I wanted to keep explicit string literals out of the scenario. But it doesn’t make that much difference either way.
Then /^the administrator (gets|does not get) an email$/ do |status|
if status == "gets"
Then '"admin@admin.com" should receive an email'
else
Then '"admin@admin.com" should not receive an email'
end
end
Everything here passes for the normal case. So far, my main note is that all the step definitions are really simple — it’s almost impossible to misinterpret them.
At this point, I move on to the blocking case. There’s one more step definition to write. In actuality, the “then 5 messages are sent” step will fail, since the blocking isn’t in the code. So, I would normally jump to writing regular tests at that point, but since I’m here, I’ll present the last step definition.
Cucumber is a black box, therefore in order to detect if the user is blocked, we need to find some place in the application that will display it. After their initial notification, the user doesn’t see anything confirming their block status. But the administrator does, via an admin screen that actually hasn’t been written yet. So, the cucumber tests logs the user out, logs an admin in, and checks the admin page for a row associated with the user.
Then /^I am blocked from sending further messages$/ do
@sender = @user
visit "/logout"
Given "I am a logged in administrator"
@admin = @user
visit path_to("the admin messaging page")
assert_select("tr#?", dom_id(@sender, :blocked_row), :count => 1)
end
This step definition is a bit more complicated than the others, and it’s also taking it slightly on faith that the row with the correct DOM id will actually have the information the admin needs. (The tradeoff in view testing, as always is faith an flexibility vs. certainty and brittleness…)
Okay, there are genuinely failing Cucumber steps, it’s time to make them pass.
A couple of things to note:
The basic ideas is write the tests in the class where the code is going to go. I can guess that code will need to go in the MessageController
, the Message
model, and the User
model. In fact, I’m going to need a new field in the user model, a string representing message sending status. (It’s a string and not a boolean, because I know from a later requirement that there will be more than two states). I actually will wait to write the migration, though, until a test compels it.
My preference is to start with the controller tests — it’s the easiest way into the system for me.
Here’s the batch of tests I wrote to cover the normal sending a message and not getting blocked functionality. These tests will probably look weird to just about everybody since I wrote them using a combination of Shoulda, Zebra, and Matchy. Zebra gives nice one-line tests, and Matchy gives a sort-of RSpec syntax that I sometimes like.
context "POST CREATE" do
setup do
ActionMailer::Base.deliveries.clear
@recipient = Factory.create(:user)
@user = login!
end
context "with a clean user" do
setup do
post :create, :recipient => @recipient.id,
:message => Factory.attributes_for(:message,
:sender_id => @user.id)
end
expect { @user.messages_sent.size.should == 1 }
expect { @user.spam_message_count.should == 1 }
expect { assigns(:message).should_not be_new_record }
expect { @user.should_not be_message_sending_blocked }
expect { assert_no_email_to_administrator }
should "create and send" do
assert_sent_email do |email|
email.to.first == @recipient.email &&
email.from.first == "do_not_reply@singlestravelintl.com"
end
end
end
### Outer context continues
What to say about these tests…
login!
method creates a factory user and simulates a login.expect
blocks each resolve into a test that passes if the block returns true. The first one checks that a message has actually gone into the database (messages_sent
is an association on user). The spam_message_count
is what is used to determine if a user is blocked — that’s not going to pass yet, because the concept of a spam count is new to the app. The third tests that the message assigned in the controller is actually saved. The fourth checks the message_sending_blocked?
method of a user, which is also going to need to be written. The last calls a helper method to determine if an email is sent to the administrator, which should only happen if the user is blocked.Many of these tests pass as is. The ones that don’t are dependent on the new user features. Which means we need user tests.
I realize that some of you are starting to think that having all these tests is ridiculous overkill — I mean, you’ve got your Cucumber tests, you’ve got your controller tests, and now model tests. That’s, like, triple the work, isn’t it?
No, I don’t think it is. My rationale for doing all this testing goes something like this:
Anyway, the user tests cover the case where a user has four messages, and then the transition to five messages. The setup is pretty similar (another reason why the extra tests don’t take as long to write as you might think)
context "rate limiting" do
setup do
ActionMailer::Base.deliveries.clear
Timecop.freeze(Date.today)
@user = Factory.create(:user, :created_at => 1.month.ago)
4.times do |i|
Message.new_from_params(
Factory.attributes_for(:message,
:created_at => i.hours.ago, :body => "Message #{i + 1}"),
@user, Factory.create(:user)).save!
end
end
should "correctly count messages" do
@user.spam_message_count.should == 4
@user.update_message_block_status.should be_nil
@user.should be_able_to_send_messages
assert_number_of_emails_to_administrator(0)
end
should "move user to blocked mode after fifth message" do
Message.new_from_params(Factory.attributes_for(:message,
:created_at => 10.minutes.ago), @user,
Factory.create(:user)).save!
@user.update_message_block_status.should be("blocked")
@user.should_not be_able_to_send_messages
assert_number_of_emails_to_administrator(1)
end
## Outer context continues
Most of this should be clear given all my blabbering so far (and I have no real clear reason for abandoning Zebra in this section of tests). One thing is that the method update_message_block_status
is actually responsible for changing the user status after a message is sent.
This is getting way long, so back next with with how this played out with new requirements, and a kind of interesting bug.
Quibbles with my testing style or process should go in the comments.
Related Posts
Article
Case Study: Capturing Quality Images for ML Algorithm
Article
Climbing the Mountain of Regulatory Documentation for SaMD
Article
5 Keys to Integrating UX Design With Agile for SaMD
Article
You Had Me at Validation