My guess is, since I was setting the current font to my own font that I loaded, it was releasing the default font, which would cause it to get deleted, even though I pulled out the default font and stored it in a variable before I did this. The font system didn't know I did this. So by using AddRef() to the default font I stored in a local variable, it would keep it alive.
I changed it to the following and it worked. Notice the AddRef() and Release()
local font = context:GetFont()
-- self.font is the font I load in Start()
font:AddRef()
context:SetFont(self.font)
context:SetBlendMode(Blend.Alpha)
context:DrawText(self.msg, pos.x, pos.y)
context:SetBlendMode(Blend.Solid)
context:SetFont(font)
font:Release()