Groovy, reusable UI widgets and variable object graphs

Sometimes Groovy is just so sweet.

sweets

So, I’m working on some new functionality in a Grails app and I wanted to reuse a previously created widget.  For purposes of this discussion, a widget is just a snippet of code for a UI control such as a dropdown box.

That widget was actually two widgets packaged as one:  it was one dropdown dependent on, or chained to, another dropdown.  If I change the value in the first dropdown, the available values in the second dropdown change.  That functionality is driven by some simple jQuery and is easily used via <g:render>.

But in the course of trying to reuse it, I hit upon an interesting dilemma:

  • in one context, the object graph had 3 levels (Asset >> Unit >> Agency)
  • in a second context, the object graph had only two levels (Unit >> Agency)

(Slightly less problematic was that in one context, the Agency dropdown was a required field and in all other contexts, it was optional.  That’s why the requiredAgency variable was used — to drive whether to display a “*” next to the dropdown label).

Here’s the resulting solution:

<div>
  <label for="agency">
    <g:message code="agency.label"/><g:if test="${requiredAgency}"><span>*</span></g:if>
  </label>
  <div>
    <div id="agencyDiv">  %{--Needed for IE9--}%
    <g:select id="agency"
              name="agency.id"
              from="${Agency.list()}"
              optionKey="id"
              optionValue="name"
              value="${parentInstance?.agency?.id}"
              noSelection="${['null':'--Select One--']}"
    />
    </div>
  </div>
</div>

The “trick”, which is really elegant in its simplicity, is to pass in a parentInstance variable to the dropdown.  If present, it’ll make use of it.  If not passed in, it’s null.  And if it’s null, this piece…

value="${parentInstance?.agency?.id}

…will cause agency to just “slide to the left”.

Here’s how it’s called in a context where I do use parentInstance:

<g:render template="/includes/widgets/chainedAgencyUnitDropdownsUsingUnit" 
          model="${[parentInstance: assetInstance?.unit, theInstance: assetInstance]}"/>

And here’s how it’s called without parentInstance:

<g:render template="/includes/widgets/chainedAgencyUnitDropdownsUsingUnit" 
          model="${[theInstance: employeeInstance, requiredAgency: true, requiredUnit: true]}"/>

Now, I’ve used the ?. operator a ton of times.  It’s indispensable.  I cry when I have to work on a raw Java project and do some silly Utils.isNullOrBlank(object) call. But I guess I never considered how the ?. operator would act when governing the very first object. I didn’t expect only that portion to “disappear” while leaving the rest of the line to be evaluatable.

The more I use Groovy the more attached I get to it.  It’s so brilliant in terms of usability.

Using Transients for Validations in Grails

Craig Burke (@craigburke1) had a tweet on July 16 that mentioned Grails transients and a mini-conversation ensued.  I mentioned that transients come in handy for “faux validations” (my term; I literally made it up on the spot) but couldn’t explain in enough clarity what I was doing with transients in 140 chars on Twitter.  This post will try to explain things.

Summary

A transient comes in handy to validate a field that won’t be saved in Object A, but needs to be saved in Object B, and Object B doesn’t have a form for editing.  It needs to be valid for Object B, so validate it on the form for Object A, but just don’t save it as part of Object A.

Details

About a week before that Twitter conversation, transients came in very handy for me.  The timing was uncanny, hence my tweets engaging him on it.  For purposes of this post “faux validations” are standard validations against a domain property via the constraints closure, but where the property is never saved.  It’s fake.

This is a piece of my constraints closure, with actual comments for the next dev behind me that has to maintain this app.

static constraints = {
    /*This style of validation is needed for Effective Date because it's transient.
      We validate it, we just don't save it (the saving will be done at the assetTransaction level). By the time we get there, we want
      a valid date to use.  We can't validate there because by that time the assetInstance has been saved and a
      validation round trip is not possible. This situation arose when we added an AssetChangedEvent and Effective Date was requested to be
      on the form. Effective Date doesn't exist on the Asset, hence it's transient.
     */
    effectiveDate(nullable: false, bindable: true, validator: { val, obj ->
                      //only validate the effectiveDate field when the entity is not yet persistent.
                      obj.id || val } )

Per the comments there, here’s the contextual situation of how transients for validation came in handy.

  • There was a form that managed CRUD for an Asset.  Nothing special here.
  • The Asset domain object does not have a property called Effective Date.  This will be important and where the transient comes in.
  • An Asset hasMany Asset Transactions.
  • An AssetTransaction belongsTo a number of different Asset Events, but for purposes of this conversation, it belongs to AssetChangedEvent.  Other Asset Events include AssetMovedEvent, AssetTransferredEvent, AssetCheckedOutEvent, and so on.
  • A recent request for an Audit Trail (I didn’t use the plugin) means I am adding potentially multiple AssetTransaction records for a single AssetChangedEvent.

To recap:  I can edit an Asset, change a bunch of fields (say, 7 fields) and have 7 entries in the Audit Trail/Transaction History.  Hold that thought.

Here’s the kicker:  For the other Asset Events, it’s not a pure CRUD situation, per se.  It’s not like you’re editing an existing record.  You’re performing an event (i.e. you’ll move an Asset or Retire an Asset, etc.)  In the case of changing an Asset, you’re doing both:  you’re editing the underlying Asset domain object, and you’re indirectly creating AssetTransaction records.

With that context, the technical point is this: Effective Date is a required field, and it needs to be a valid Date.  That’s easy…but remember that form doing the Asset editing is not binding to the AssetTransaction domain object — it’s binding to the Asset domain object.

That’s where the transient comes in.  We attach a transient EffectiveField property to Asset and now the constraints for it are used in validations.  Since the Asset won’t save unless validations pass, we know that when it does save, EffectiveDate is a valid Date.  With that field in hand, once a few other things happen, we eventually take that value and attach it to the AssetTransaction.effectiveDate field.  And because we might have multiple AssetTransactions per edit (one per field changed), we’ll stick that single EffectiveDate onto each AssetTransaction.effectiveDate like so:

//Update AssetTransaction Record
changesMade.each {
    def fieldChanged = it.key
    def originalValue = it.value.originalValue
    def newValue = it.value.newValue

    def assetChangedEventInstance = new AssetChangedEvent(params)
    AssetTransaction assetTransaction = new AssetTransaction()
    assetChangedEventInstance.assetTransaction = assetTransaction
    assetChangedEventInstance?.assetTransaction?.effectiveDate = assetInstance?.effectiveDate
    assetChangedEventInstance?.assetTransaction?.transactionType = AssetTransactionTypeEnum.CHANGED.key
    assetChangedEventInstance?.assetTransaction?.note = "Field $fieldChanged changed:  from [$originalValue] to [$newValue]"
    assetChangedEventInstance?.assetTransaction?.asset = assetInstance
    assetChangedEventInstance.accessControlGroup = assetInstance.accessControlGroup
    assetChangedEventInstance?.assetTransaction?.accessControlGroup = assetInstance.accessControlGroup
    assetChangedEventInstance.save(flush: true)
}

This functionality was just added a few days ago and I’m sure it could be more elegant.  It feels a little “bulky” or verbose for Groovy, but it works.

It’s also important to mention that transients don’t automatically bind, so the bindable constraint had to be set:

effectiveDate(nullable: false, bindable: true, validator: { val, obj ->
          //only validate the effectiveDate field when the entity is not yet persistent.
          obj.id || val } )

And, in the interest completeness, here’s the effectiveDate field in the Asset object.

String assetTagGroup                //Transient
String assetTagRangeStart           //Transient
String assetTagRangeQty             //Transient
Date effectiveDate = new Date()     //Transient

static transients = ['assetTagGroup', 'assetTagRangeQty', 'assetTagRangeStart', 'effectiveDate']

That’s it.

Grails findAllBy and findAll, and the impact of your primary key on them

So I’m working on a couple of REST API calls, exposing some underlying DB data for a project.  About 15 successful API calls into it, I hit a wall: the amount of data being returned doesn’t mesh with what’s in the DB.

I did the obvious. I jotted down expected counts of records that should be returned by running raw SQL. This is nothing elaborate; I was just using a single field in the WHERE clause.

Here’s the table in question:

class CommunityCalendarEvents {
  Integer calendarId
  Date eventStartDate
  Date eventEndDate
  String eventName
  String eventDetails
  String eventURL
  String contactDetails
  String location
  Date modifiedDate

  static mapping = {
    table name:("PORTAL_CALENDAR_EVENTS")
    version false
    id generator:"assigned", name:"calendarId"
    cache usage:"read-only"

    columns {
      calendarId column: "CALENDAR_ID", sqlType: "NUMBER"
      eventStartDate column: "EVENT_START_DATE", sqlType: "DATE"
      eventEndDate column: "EVENT_END_DATE", sqlType: "DATE"
      eventName column: "EVENT_NAME", sqlType: "VARCHAR2(256 BYTE)"
      eventDetails column: "EVENT_DETAILS", sqlType: "VARCHAR2(512 BYTE)"
      eventURL column: "EVENT_URL", sqlType: "VARCHAR2(512 BYTE)"
      contactDetails column: "CONTACT_DETAILS", sqlType: "VARCHAR2(400 BYTE)"
      location column: "LOCATION", sqlType: "VARCHAR2(512 BYTE)"
      modifiedDate column: "MODIFIED_DATE", sqlType: "DATE"
    }
  }
}

This maps to a legacy table in Oracle.  Importantly, that table is not the source of record but itself a copy of some other table (of which I’m not aware of).  So this table is populated via some push operation that occurs on a regular basis.  Whether this is a best practice or not is somewhat irrelevant — these are the cards I’m dealt. I have no say in this matter.

Using this structure and mapping, if I run a findAllByCalendarId(5), I get…..1 record.  There are 38 of them.  I’m scratching my head.  This can’t be.  So I dig some more.

I try a countByCalendarId(5) and I get…38.  Now, I’m perplexed.  Grails is smart enough to know there are 38 records that fit the condition, but it won’t give em to me with the findAllBy().

At this point I’m trying everything, including rebooting the laptop (seriously).  I’d light a candle for Saint Grails if I had one. This is way too weird.  Nothing, and I mean nothing, I did could fix this problem. I try the findAllBy() on some other tables and they give me the proper amount of records.  I’m dying here.

128-the-agony-gethsemane

I experiment some more.  I try findAll(). I’m expecting some 3600+ records in the results.  I get 24.  Not 2400.  24.  Just kill me.

Something is very broken and it affects only this one table.  I pour over the mapping and I can’t see what could be wrong.  Then serendipity hits. By chance I’m examining the stats on that table in Oracle SQL Developer and I see something way too interesting. I noticed that it reported 24 unique records.  That number 24 just jumped out at me.  I saw it earlier.  At least now I’m seeing some sense of order, even if I don’t understand it yet.  At least it’s not just some freak adhoc situation.

And then…I hit the motherlode.  While researching this online, I stumble across some poor guy that hit the same exact situation as me 4 years ago:

http://grails.1312388.n4.nabble.com/FindAllWhere-returns-only-one-object-Bug-td1379750.html

And the answer hits me.  This is the one table that doesn’t have a PK.  This is why the findAllBy() works on other tables but fails on this one.  When Burt Beckwith asks “I’m confused – id is the primary key, right? How can you have multiple rows with the same primary key?”…I knew what had to be done.

This one line is not going to work:

id generator:"assigned", name:"calendarId"

This is why I get one record.

That mapping is my fault, and I’ll own up to it, but I will argue that this is broken in Grails.  Badly.  Beyond badly.  When I ask for all records, I want…all records.  I don’t want you examining the PK to determine how many records are in the DB.  The number of records returned by findAllBy() must equal countBy().  The number of records returned by findAll() must equal count().  There’s probably some optimization going on underneath, but that optimization broke the Principle of Least Surprise.  Trust me…I was surprised to see 38 records magically become 1 record, or 3600+ records magically become 24 records.

So how do you fix it?  With a hack, of sorts.  I would prefer some way to define a domain object without a PK, but that doesn’t seem to be an option (as of Grails 2.2).  Someone on StackOverflow mentioned to someone’s question with something along the lines of “you should really think about your design if you have tables without a PK”.  He should really think outside the box and consider not all projects are greenfield projects and sometimes you might have to play in someone else’s world and they have tables without a PK. For whatever reason, they do. Off the top of my head, denormalized tables for reporting purposes and for which there are no transactions run on the table are rather common.

Anyways, I needed a faux PK.  I needed something so unique that it effectively becomes a PK. Checking what’s non-nullable, I had few options available to me but this seems to work:

id composite: ['calendarId', 'eventStartDate', 'eventName']

That does it.  A composite PK needed only and exclusively to correct a flaw in how findAllBy() and findAll() work in Grails.

#smh…

Grails deployments to JBoss 6.1 and Tomcat 7

I had a situation where the Grails app I was building needed to be deployable to JBoss 6.1 as well as Tomcat 7.

This has to be one of my favorite snippets of code (in BuildConfig.groovy):

dependencies {
    //Added to deal with problems running Quartz on JBoss
    //NoSuchMethodError: org.objectweb.asm.ClassWriter.<init>(I)V]
    compile ('cglib:cglib-nodep:2.2')
}

grails.war.resources = { stagingDir, args ->
    //Added to deal with problems running Quartz on JBoss
    //NoSuchMethodError: org.objectweb.asm.ClassWriter.<init>(I)V]
    println "Removing export jars from WEB-INF/lib/"
    delete(file: "${stagingDir}/WEB-INF/lib/cglib-2.2.jar")
}

That was added to deal with some problems that arose when using the Quartz 0.4.2 plugin.  I wouldn’t see the problem on localhost development, but only when deploying to JBoss 6.1.

The beauty of it is that it fixes the JBoss issue, but Tomcat ignores it. Jetty ignores it too.

ignore

You know what time it is…

If you’re working remotely on a Grails app, do yourself a favor and save your sanity.  I just completely a nasty debugging session where I’ve got a Where query that was filtering on a date and I was getting more records than I should have been.

I was digging under every rock, but I couldn’t fix it.  (No, Joda Time didn’t help, I dug under that rock too).

Then I stared at the results and I was always off by the same amount:  8 hours.

That was way too odd.  For any given date, I was bringing 8 hours more data than what I asked for.  Time zones.  It had to be time zones.

But here’s the kicker:  I’m working in Huntington Beach, connecting to a database in Chicago, which was set to London time.  Where is that glass of wine?

One line of code fixes this madness:

TimeZone.setDefault(TimeZone.getTimeZone("Europe/London"))

Stick that bad boy in your BootStrap init closure and it acts like penicillin.  It just…fixes…everything.

(Well, don’t necessarily set it to London time, but you get the idea.  Set it to whatever the database server is set to).

Trials and tribulations with the left-join and pagination in Grails

(This post, indeed this entire blog, was started to share the details of something I was working on and @danveloper wanted more details on it.)

So I’m in bug fixing mode and finally got back to something I discovered months ago (the project is not yet live), but never had a chance to dig in since it was fairly low priority.  But it annoyed me.

The issue was related to sorting a table returned by list().  Normally, this is a freebie.  Pass in params.sort and everything just works.  And it did, for all columns except two.  When I say all columns, I mean all columns in all my tables in the entire app…except two.  This is why it annoyed me.  It just seemed odd.

To cut to the chase, those two columns (Unit and Employee) were nullable One-to-One associations to a domain object called Asset. About an hour into looking into this, that fact turned out to be the key fact.

For sake of discussion, the Asset table has about 3200 records on a localhost dev DB. When you arrive at the asset/list page, you’ll see all the usual elements:

  • a total record count
  • a table showing the first 100 records (in my case)
  • pagination at the bottom

It turned out that list() was performing an inner-join. And that’s fine, even if you have null associations on some records.  But, it turns bad when you chose to sort on a nullable association, such as Unit or Employee in my case.

At that point, my total record count was accurate, my table results are “accurate”, but my pagination is all off because it was based on the record count.  I put quotes around “accurate” because, it’s accurate, but not what I wanted.  I wanted what amounts to a left join:  show me all Assets, and if the Asset is assigned to a Unit or checked out to an Employee, then show that too.  But for God’s sake, show me all the Assets.  Do not remove them just because they are not assigned to a Unit or checked out to an Employee.

list() won’t do that.  Not the typical way, anyway.  More on that in a second.

Given my data, if I had 3 Assets assigned to a Unit and I chose to sort on Unit, I would see…..(drumroll)…..3 records.

Yes, technically “accurate” but lame.  I’m showing 3 records in the table, a total record count of 3200, and pagination to match the 3200 records.  Why the pagination disconnect?  Because I’m not using size() which has performance implications.  I do a separate countBy() on Assets to get that total.  And the countBy() is truly accurate.  And fast.  But since pagination is using that value, I’ve got pagination buttons down below that will say “Page 31″….for three records.

Pass me a glass of wine, I’m thinking.  You can’t be serious.  Really?

I experimented with using a left join via createCriteria and createAlias, but that was ridiculously slow.  Either that, or I didn’t implement it correctly.  I had YourKit available and I timed each case:  using the list() in the normal way and using list() with createCriteria and createAlias for the left join.  I knew it was slower.  I could feel it when navigating to that page.  It was noticeable.  But I wanted data. To profile, I bounced the server prior to each measurement to ensure apples-to-apples comparisons, no caching, etc.

Here are those profiling snapshots:

Left Join

Left Join

Inner Join

Inner Join

When I saw this, I immediately decided to change course.  Again, with other things on my plate, I simply didn’t have the time to mess around and try to tweak this in some way to get back to a reasonable level of performance. It wasn’t twice as slow.  It was slower by an order of magnitude.  And it affected performance on first load and on sorting.  Basically, it was just horrible.

The only upside it had was that it would give me my results in the way I intrinsically expected:  show me all the Assets and don’t remove them just because of null associations, even when I sort.

I had to compromise.  After thinking about this in the shower, I’m still not satisfied with the overall solution, but it’s a workable solution (I won’t have 3 records show up on a sort with pagination for 3200 records).

The solution was to keep the inner join and all the speed associated with that, but changing how I calculate total records.  Once that number changes, pagination becomes accurate again.

Here’s what that looks like:

        if (params.sort == "unit.unitNumber" ){
            assetInstanceTotal = accessControlGroups.size() > 0? Asset.countByAccessControlGroupInListAndUnitIsNotNull(accessControlGroups):0
        }  else if (params.sort == "employee.firstName") {
            assetInstanceTotal = accessControlGroups.size() > 0? Asset.countByAccessControlGroupInListAndEmployeeIsNotNull(accessControlGroups):0
        } else {
            assetInstanceTotal = accessControlGroups.size() > 0? Asset.countByAccessControlGroupInList(accessControlGroups):0
        }

In my gut, I’m still not sold. I don’t like it.  Had it affected more tables, I would have spent more time looking into why I was getting abysmal performance from the left join. It was painfully slow on 3200 records.  The actual database has 115,000+ records.  I just wasn’t in the mood to fight this battle right now when a reasonable solution was available.

If I had the createCriteria/createAlias implementation wrong, I’d love to know what the proper way is.  Please chime in.  I’ll probably continue experimenting with this in my off hours out of curiosity.   I just can’t devote work time to it.  Something inside tells me this is not an obscure use case, and that Grails/GORM should handle this in a more elegant way.