Error Retrieving Mail from Gmail with JavaMail + Scala

Question

I'm trying to use Scala and JavaMail + gimaps (Gmail Extensions). I'm currently using v1.5.1 of both, scala 2.10.3, java JDK 1.7.0_45. Things almost work, I can get a GmailSSLStore, I can load a GmailFolder into a variable called allMail.

However, when I call allMail.getMessages(), instead of returning GmailMessage's like I'd expect, it's returning normal IMAPMessages. Any idea why this is happening? I stepped through the code, and it looks like perhaps the GmailProtocol isn't correctly being used, but I have no idea why that would be the case. Any help would be appreciated!

This is in Akka, BTW.

  var store: GmailStore = _
  var allMail: GmailFolder = _

  override def preStart = {
    val props = System.getProperties()
    props.put("mail.debug", "true")
    props.put("mail.store.protocol", "gimaps")

    val session = Session.getDefaultInstance(props, null)
    store = session.getStore("gimaps").asInstanceOf[GmailStore]
    store.connect("test@gmail.com", "Password" )
    val folders = store.getDefaultFolder.list("*").map(m => m.asInstanceOf[GmailFolder])
    allMail = folders.filter(m => m.getAttributes.contains("\\All")).head
    allMail.open(Folder.READ_ONLY)
  }

//Some Code Elided

def getMail() = {
   val ms = allMail.getMessages()
   val fp = new FetchProfile()
   fp.add(GmailFolder.FetchProfileItem.MSGID)
   fp.add(GmailFolder.FetchProfileItem.THRID)
   fp.add(GmailFolder.FetchProfileItem.LABELS)
   allMail.fetch(ms, fp)
   val messages = ms.map(m => {
      m.asInstanceOf[GmailMessage]  // <--- This line crashes, m is IMAPMessage
   })
}

The output is the following. One note, I just see what might be a problem, why is it listing JavaMail 1.4.4 there? The dependencies are pretty clearly setup in Built.SBT

  "com.sun.mail" % "javax.mail" % "1.5.1",
  "com.sun.mail" % "gimap" % "1.5.1"

Console Output:

DEBUG: JavaMail version 1.4.4
DEBUG: URL jar:file:/C:/Users/Patrick/.ivy2/cache/com.sun.mail/gimap/jars/gimap-1.4.7.jar!/META-INF/javamail.providers
DEBUG: successfully loaded resource: jar:file:/C:/Users/Patrick/.ivy2/cache/com.sun.mail/gimap/jars/gimap-1.4.7.jar!/META-INF/javamail.providers
DEBUG: URL jar:file:/C:/Source/GIT/Core/sync/lib/gimap-1.5.1.jar!/META-INF/javamail.providers
DEBUG: successfully loaded resource: jar:file:/C:/Source/GIT/Core/sync/lib/gimap-1.5.1.jar!/META-INF/javamail.providers
DEBUG: successfully loaded resource: /META-INF/javamail.default.providers
DEBUG: Tables of loaded providers
DEBUG: Providers Listed By Class Name: {com.sun.mail.smtp.SMTPSSLTransport=javax.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Sun Microsystems, Inc], com.sun.mail.smtp.SMTPTransport=javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Sun Microsystems, Inc], com.sun.mail.imap.IMAPSSLStore=javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Sun Microsystems, Inc], com.sun.mail.gimap.GmailSSLStore=javax.mail.Provider[STORE,gimaps,com.sun.mail.gimap.GmailSSLStore,Oracle], com.sun.mail.pop3.POP3SSLStore=javax.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Sun Microsystems, Inc], com.sun.mail.imap.IMAPStore=javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Sun Microsystems, Inc], com.sun.mail.gimap.GmailStore=javax.mail.Provider[STORE,gimap,com.sun.mail.gimap.GmailStore,Oracle], com.sun.mail.pop3.POP3Store=javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Sun Microsystems, Inc]}
DEBUG: Providers Listed By Protocol: {imaps=javax.mail.Provider[STORE,imaps,com.sun.mail.imap.IMAPSSLStore,Sun Microsystems, Inc], imap=javax.mail.Provider[STORE,imap,com.sun.mail.imap.IMAPStore,Sun Microsystems, Inc], gimap=javax.mail.Provider[STORE,gimap,com.sun.mail.gimap.GmailStore,Oracle], smtps=javax.mail.Provider[TRANSPORT,smtps,com.sun.mail.smtp.SMTPSSLTransport,Sun Microsystems, Inc], pop3=javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Sun Microsystems, Inc], pop3s=javax.mail.Provider[STORE,pop3s,com.sun.mail.pop3.POP3SSLStore,Sun Microsystems, Inc], smtp=javax.mail.Provider[TRANSPORT,smtp,com.sun.mail.smtp.SMTPTransport,Sun Microsystems, Inc], gimaps=javax.mail.Provider[STORE,gimaps,com.sun.mail.gimap.GmailSSLStore,Oracle]}
DEBUG: successfully loaded resource: /META-INF/javamail.default.address.map
DEBUG: getProvider() returning javax.mail.Provider[STORE,gimaps,com.sun.mail.gimap.GmailSSLStore,Oracle]
DEBUG: mail.imap.fetchsize: 16384
DEBUG: mail.imap.statuscachetimeout: 1000
DEBUG: mail.imap.appendbuffersize: -1
DEBUG: mail.imap.minidletime: 10
DEBUG: trying to connect to host "imap.gmail.com", port 993, isSSL true
* OK Gimap ready for requests from 173.164.155.206 zl9mb39448964pac
A0 CAPABILITY
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN
A0 OK Thats all she wrote! zl9mb39448964pac
DEBUG IMAP: AUTH: XOAUTH
DEBUG IMAP: AUTH: XOAUTH2
DEBUG IMAP: AUTH: PLAIN
DEBUG IMAP: AUTH: PLAIN-CLIENTTOKEN
DEBUG: protocolConnect login, host=imap.gmail.com, user=test@gmail.com, password=<non-null>
A1 AUTHENTICATE PLAIN
+ 
AHRlc3RAc3luYXRhLmNvbQBTeW5hdGExIQ==
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH
A1 OK test@gmail.com Test Account authenticated (Success)
A2 CAPABILITY
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH
A2 OK Success
A3 LIST "" "*"
* LIST (\HasNoChildren) "/" "INBOX"
* LIST (\Noselect \HasChildren) "/" "[Gmail]"
* LIST (\HasNoChildren \All) "/" "[Gmail]/All Mail"
* LIST (\HasNoChildren \Drafts) "/" "[Gmail]/Drafts"
* LIST (\HasNoChildren \Important) "/" "[Gmail]/Important"
* LIST (\HasNoChildren \Sent) "/" "[Gmail]/Sent Mail"
* LIST (\HasNoChildren \Junk) "/" "[Gmail]/Spam"
* LIST (\HasNoChildren \Flagged) "/" "[Gmail]/Starred"
* LIST (\HasNoChildren \Trash) "/" "[Gmail]/Trash"
A3 OK Success
DEBUG: connection available -- size: 1
A4 EXAMINE "[Gmail]/All Mail"
* FLAGS (\Answered \Flagged \Draft \Deleted \Seen $Phishing $NotPhishing)
* OK [PERMANENTFLAGS ()] Flags permitted.
* OK [UIDVALIDITY 11] UIDs valid.
* 4 EXISTS
* 0 RECENT
* OK [UIDNEXT 590] Predicted next UID.
* OK [HIGHESTMODSEQ 148419]
A4 OK [READ-ONLY] [Gmail]/All Mail selected. (Success)
DEBUG IMAP: IMAPProtocol noop
A5 NOOP
A5 OK Success
A6 CLOSE
[ERROR] [02/21/2014 09:37:38.404] [ConnectionSystem-akka.actor.default-dispatcher-4] [akka://ConnectionSystem/user/MainConnectionManager/googleAppsConnectionManager-5305184d4500004500e8fe66/gmailManager/$a] com.sun.mail.imap.IMAPMessage cannot be cast to com.sun.mail.gimap.GmailMessage
java.lang.ClassCastException: com.sun.mail.imap.IMAPMessage cannot be cast to com.sun.mail.gimap.GmailMessage
    at com.test.sync.google.GmailSyncer$$anonfun$firstSync$1$$anonfun$2.apply(GmailSyncer.scala:90)
    at com.test.sync.google.GmailSyncer$$anonfun$firstSync$1$$anonfun$2.apply(GmailSyncer.scala:89)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
    at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108)
    at com.test.sync.google.GmailSyncer$$anonfun$firstSync$1.applyOrElse(GmailSyncer.scala:89)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
    at akka.actor.ActorCell.invoke(ActorCell.scala:456)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
    at akka.dispatch.Mailbox.run(Mailbox.scala:219)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

A6 OK Returned to authenticated state. (Success)
DEBUG: added an Authenticated connection -- size: 1
A7 LOGOUT
* BYE LOGOUT Requested
A7 OK 73 good day (Success)
DEBUG: IMAPStore connection dead
DEBUG: IMAPStore cleanup, force false
DEBUG: IMAPStore cleanup done

Answer 1

Ok, we finally for real figured this out, I hope this can help somebody else.

Basically, Lift has a transitive dependency on Java Mail 1.4.4, HOWEVER, it's on a different packaging than we were referencing (javax.mail, javamail, 1.4.4 vs sun.com.mail, javax.mail, 1.5.1). So, I'd almost classify this as a bug, but there was overlapping Jars with overlapping package names, but SBT wasn't recognizing it as a true dependency conflict. Depending on the mood of the JVM, either the 1.4.4 jar or the 1.5.1 jar would be loaded, meaning every now an again we'd see NoMethodFound exceptions when we thought we were calling the 1.5.1 library, but in fact the 1.4.4 library was being loaded.

We debugged this by just searching through the dependency resolution logs, but we also learned after we could use the ivy dependency report to see what library had a dependency on the old Java Mail component.

Answer 2

I'm guessing that you have both JavaMail 1.4.4 and JavaMail 1.5.1 in your classpath. Even though it's finding the "gimaps" provider from JavaMail 1.5.1, it's using the IMAP classes from 1.4.4, and those classes don't have the hooks that allow the gimaps provider to extend it and provide Gmail-specific support.