CODEWITHSUNDEEP
mongodb
helion3
helion3

Reputation: 37381

Mongo TTL not removing documents

I'm toying with auto-expiring documents from a collection. The java application creates an index per the Mongo TTL docs.

coll.createIndex(new Document("Expires", 1).append("expireAfterSeconds", 0));

When inserting my document, I set the Expires field to a future Date. For this testing I've been setting it 1 minute in the future.

I've verified the date exists properly, the index appears to be correct, and I've waited 10+ minutes (even though the ttl runner operates every sixty seconds) but the document remains.

{
    "_id" : ObjectId("569847baf7794c44b8f2f17b"),
    // my data
    "Created" : ISODate("2016-01-15T02:02:30.116Z"),
    "Expires" : ISODate("2016-01-15T02:03:30.922Z")
}

What else could I have missed? Here are the indexes:

[
    {
        "v" : 1,
        "key" : {
            "_id" : 1
        },
        "name" : "_id_",
        "ns" : "prism.prismEventRecord"
    },
    {
        "v" : 1,
        "key" : {
            "Location.X" : 1,
            "Location.Z" : 1,
            "Location.Y" : 1,
            "Created" : -1
        },
        "name" : "Location.X_1_Location.Z_1_Location.Y_1_Created_-1",
        "ns" : "prism.prismEventRecord"
    },
    {
        "v" : 1,
        "key" : {
            "Created" : -1,
            "EventName" : 1
        },
        "name" : "Created_-1_EventName_1",
        "ns" : "prism.prismEventRecord"
    },
    {
        "v" : 1,
        "key" : {
            "Expires" : 1,
            "expireAfterSeconds" : 0
        },
        "name" : "Expires_1_expireAfterSeconds_0",
        "ns" : "prism.prismEventRecord"
    }
]

Upvotes: 1

Views: 3071

Answers (2)

Eduardo A del Corral Lira
Eduardo A del Corral Lira

Reputation: 89

In case you run into this question as a Golang user, you have two choices:

1 Use structures: This works when you know the payload structure, and is documented extensively

2 Introduce an actual date object into your JSON payload: Only use this if for some reason your payload structure absolutely can't be known ahead of time for some reason.

In my case, the source data comes from a system whose structure is a black-box. I first tried to introduce an ISO-compliant TTL index whose format matched Mongo's but it was still read as text. This led me to deduct that it was due to the driver not being instructed to format it properly.

I believe a deep dive into Mongo's Golang driver's specifics to manipulate this process could only give us a short lived solution (since the implementation details are subject to change). So instead, I suggest introducing a real date property into your payload and let the driver adapt it for Mongo (adapting the principle in this snippet to your own code structure)

err = json.Unmarshal([]byte (objectJsonString.String()), &objectJson)
if err == nil && objectJson != nil {
        //Must introduce as native or they are considered text
        objectJson["createdAt"] = time.Now()
        //Your other code
        insertResult, err := collection.InsertOne(context.TODO(), objectJson)
}

So basically, create your JSON or BSON object normally with the rest of the data, and then introduce your TTL index using real date values rather than attempt to have your JSON parser do that work for you.

I look forward to corrections, just please be civil and I'll make sure to update and improve this answer with any observations made.

Upvotes: 0

bri
bri

Reputation: 3030

I wonder if it makes sense to take the java mongo client out of the pic for a minute.

I have created a similar collection, and made the following call in the shell.

db.weblog.createIndex({"expireAt":1},{expireAfterSeconds:0})

When I do, and then I call db.weblog.getIndexes(), this is what the expiring index looks like:

{
    "v" : 1,
    "key" : {
        "expireAt" : 1
    },
    "name" : "expireAt_1",
    "ns" : "logs.weblog",
    "expireAfterSeconds" : 0
}

I think your java call may be "appending" a new column to your index (not setting the property you were hoping to set). Take a look... your index def looks like this:

{
    "v" : 1,
    "key" : {
        "Expires" : 1,
        "expireAfterSeconds" : 0
    },
    "name" : "Expires_1_expireAfterSeconds_0",
    "ns" : "prism.prismEventRecord"
}

See what I mean? "expireAfterSeconds is a key, not a property. Now -- how do you do THAT with the java shell? Ummm ... don't yell at me, but Im a c# guy ... I found a post or two that punt on the question of ttl indexes from the java client, but they're old-ish.

Maybe the java client has gotten better and now supports options? Hopefully, knowing what the problem is gives a guy with your stellar coding skills enough to take it from here ;-)

EDIT: Stack java driver code (untested):

IndexOptions options = new IndexOptions()
    .name('whocareswhatwecallthisindex')
    .expireAfter(1, TimeUnit.DAYS);
coll.createIndex(new Document("Expires", 1), options);

EDIT2: C# driver code to create the same index:

var optionsIdx = new CreateIndexOptions() { ExpireAfter = new TimeSpan(0)};
await coll.Indexes.CreateOneAsync(Builders<MyObject>.IndexKeys.Ascending("expiresAt"), optionsIdx);

Upvotes: 2

Related Questions