CODEWITHSUNDEEP
elasticsearchtfscode-search
Shayki Abramczyk
Shayki Abramczyk

Reputation: 41805

Retention policy to TFS Code Search Server (Elastic Search)

We have TFS 2017.3 with separate Code Search server.

We have huge TFS DB (about 1.6TB), in the code search server we have 700GB dis space.

After few weeks the disk space running out and the code search not work in the tfs.

After we increase the disk space the search back to work. How can we make retention policy to delete old code search data (index)? we don't want to increased more the disk space.

Upvotes: 1

Views: 743

Answers (2)

Shayki Abramczyk
Shayki Abramczyk

Reputation: 41805

I was able to reduce the disk size after deleting the ES folders, reinstalling the code search extension, and sometimes had to run the MissingIndexFolderTriggerCollectionIndexing.ps1.

But - I came to the conclusion that it was not worth doing, the disk size was growing rapidly and reaching the original size, so I did not save anything.

Although Microsoft recommends giving disk space of 35% of the DB, it is not enough for us and we increase the size when the disk is full to the end (currently about 45% of the DB size).

The conclusion - don't touch the ES, if the disk fills up then increase the disk size.

Upvotes: 0

PatrickLu-MSFT
PatrickLu-MSFT

Reputation: 51183

Search indexing (Code and Work Item) works in 2 phases:

  • Bulk Indexing (BI) where the entire code and work item artifacts in all projects/repositories under a Collection are indexed. This is a
    time consuming operation and depends on the size of the artifacts
    under the collection.
  • Continuous Indexing (CI) which handles all incremental updates to the artifacts (add/updated/delete) and indexes them. This is notification based model where the indexer listens to TFS events and operates based on those event notifications. CI handles almost all update operations including CRUD operations at Project/Repository/Collection layer (such as Repository renames, Project add/deletes, etc.). The operation time for these CI would depend again on the size of the incremental update. BI always precedes CI i.e. a CI will never execute on a project/repository until BI is completed for the same.

How to Clean-up Index Data and Re-index please follow below steps:

  • Pause Indexing for all collections. Run the following script on TFS Configuration DB https://github.com/Microsoft/Code-Search/blob/master/PauseIndexing.ps1
  • Login to the machine where the Elasticsearch (ES) is running
  • Stop the ES service
  • Delete the entire Search Index folder (something like, C:\TfsData\Search\IndexStore, or wherever you had configured it to be)
  • Restart the TFS Job Agent service(s) on the AT machines
  • Delete the following tables from each of the collection DBs

DELETE FROM [Search].[tbl_IndexingUnit]
DELETE FROM [Search].[tbl_IndexingUnitChangeEvent]
DELETE FROM [Search].[tbl_IndexingUnitChangeEventArchive]
DELETE FROM [Search].[tbl_JobYield]
DELETE FROM [Search].[tbl_TreeStore]
DELETE FROM [Search].[tbl_DisabledFiles]
DELETE FROM [Search].[tbl_ResourceLockTable]

More details please refer this blog in MSDN: Resetting Search Index in Team Foundation Server

Upvotes: 0

Related Questions