CODEWITHSUNDEEP
firefoxsqlite
Vitor Silva
Vitor Silva

Reputation: 17612

Programmatically access browser history

how can i create an application to read all my browser (firefox) history? i noticed that i have in

C:\Users\user.name\AppData\Local\Mozilla\Firefox\Profiles\646vwtnu.default

what looks like a sqlite database (urlclassifier3.sqlite) but i don't know if its really what is used to store de history information. i searched for examples on how to do this but didn't find anything.

ps: although the title is similar i believe this question is not the same as "How do you access browser history?"

Upvotes: 4

Views: 8742

Answers (4)

amirouche
amirouche

Reputation: 7873

Indeed the history is in places.sqlite but the database file is locked. So you need to make a copy to be able to access it:

$ pwd
/home/amirouche/.mozilla/firefox/p4x432.default
$ ls -l *sqlite
-rw-r--r-- 1 amirouche amirouche   229376 Oct  4 12:39 content-prefs.sqlite
-rw-r--r-- 1 amirouche amirouche  1572864 Oct  4 12:51 cookies.sqlite
-rw-r--r-- 1 amirouche amirouche 40501248 Oct  4 12:47 favicons.sqlite
-rw-r--r-- 1 amirouche amirouche   294912 Oct  4 12:46 formhistory.sqlite
-rw-r--r-- 1 amirouche amirouche   196608 Oct  4 12:50 permissions.sqlite
-rw-r--r-- 1 amirouche amirouche 36700160 Oct  4 12:50 places.sqlite
-rw-r--r-- 1 amirouche amirouche    65536 Oct  4 11:50 protections.sqlite
-rw-r--r-- 1 amirouche amirouche      512 Jul 24 23:41 storage.sqlite
-rw-r--r-- 1 amirouche amirouche   131072 Oct  4 12:05 storage-sync.sqlite
-rw-r--r-- 1 amirouche amirouche 15892480 Oct  4 12:51 webappsstore.sqlite

$ sqlite3 places.sqlite 
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> .schema
Error: database is locked
sqlite> 

$ cp places.sqlite places.backup.sqlite

$ sqlite3 places.backup.sqlite 
SQLite version 3.27.2 2019-02-25 16:06:06
Enter ".help" for usage hints.
sqlite> .schema

Here is the output, one interesting table is moz_places on line 2:

CREATE TABLE moz_origins ( id INTEGER PRIMARY KEY, prefix TEXT NOT NULL, host TEXT NOT NULL, frecency INTEGER NOT NULL, UNIQUE (prefix, host) );
CREATE TABLE moz_places (   id INTEGER PRIMARY KEY, url LONGVARCHAR, title LONGVARCHAR, rev_host LONGVARCHAR, visit_count INTEGER DEFAULT 0, hidden INTEGER DEFAULT 0 NOT NULL, typed INTEGER DEFAULT 0 NOT NULL, frecency INTEGER DEFAULT -1 NOT NULL, last_visit_date INTEGER , guid TEXT, foreign_count INTEGER DEFAULT 0 NOT NULL, url_hash INTEGER DEFAULT 0 NOT NULL , description TEXT, preview_image_url TEXT, origin_id INTEGER REFERENCES moz_origins(id));
CREATE TABLE moz_historyvisits (  id INTEGER PRIMARY KEY, from_visit INTEGER, place_id INTEGER, visit_date INTEGER, visit_type INTEGER, session INTEGER);
CREATE TABLE moz_inputhistory (  place_id INTEGER NOT NULL, input LONGVARCHAR NOT NULL, use_count INTEGER, PRIMARY KEY (place_id, input));
CREATE TABLE moz_bookmarks (  id INTEGER PRIMARY KEY, type INTEGER, fk INTEGER DEFAULT NULL, parent INTEGER, position INTEGER, title LONGVARCHAR, keyword_id INTEGER, folder_type TEXT, dateAdded INTEGER, lastModified INTEGER, guid TEXT, syncStatus INTEGER NOT NULL DEFAULT 0, syncChangeCounter INTEGER NOT NULL DEFAULT 1);
CREATE TABLE moz_bookmarks_deleted (  guid TEXT PRIMARY KEY, dateRemoved INTEGER NOT NULL DEFAULT 0);
CREATE TABLE moz_keywords (  id INTEGER PRIMARY KEY AUTOINCREMENT, keyword TEXT UNIQUE, place_id INTEGER, post_data TEXT);
CREATE TABLE sqlite_sequence(name,seq);
CREATE TABLE moz_anno_attributes (  id INTEGER PRIMARY KEY, name VARCHAR(32) UNIQUE NOT NULL);
CREATE TABLE moz_annos (  id INTEGER PRIMARY KEY, place_id INTEGER NOT NULL, anno_attribute_id INTEGER, content LONGVARCHAR, flags INTEGER DEFAULT 0, expiration INTEGER DEFAULT 0, type INTEGER DEFAULT 0, dateAdded INTEGER DEFAULT 0, lastModified INTEGER DEFAULT 0);
CREATE TABLE moz_items_annos (  id INTEGER PRIMARY KEY, item_id INTEGER NOT NULL, anno_attribute_id INTEGER, content LONGVARCHAR, flags INTEGER DEFAULT 0, expiration INTEGER DEFAULT 0, type INTEGER DEFAULT 0, dateAdded INTEGER DEFAULT 0, lastModified INTEGER DEFAULT 0);
CREATE TABLE moz_meta (key TEXT PRIMARY KEY, value NOT NULL) WITHOUT ROWID ;
CREATE TABLE sqlite_stat1(tbl,idx,stat);
CREATE INDEX moz_places_url_hashindex ON moz_places (url_hash);
CREATE INDEX moz_places_hostindex ON moz_places (rev_host);
CREATE INDEX moz_places_visitcount ON moz_places (visit_count);
CREATE INDEX moz_places_frecencyindex ON moz_places (frecency);
CREATE INDEX moz_places_lastvisitdateindex ON moz_places (last_visit_date);
CREATE UNIQUE INDEX moz_places_guid_uniqueindex ON moz_places (guid);
CREATE INDEX moz_places_originidindex ON moz_places (origin_id);
CREATE INDEX moz_historyvisits_placedateindex ON moz_historyvisits (place_id, visit_date);
CREATE INDEX moz_historyvisits_fromindex ON moz_historyvisits (from_visit);
CREATE INDEX moz_historyvisits_dateindex ON moz_historyvisits (visit_date);
CREATE INDEX moz_bookmarks_itemindex ON moz_bookmarks (fk, type);
CREATE INDEX moz_bookmarks_parentindex ON moz_bookmarks (parent, position);
CREATE INDEX moz_bookmarks_itemlastmodifiedindex ON moz_bookmarks (fk, lastModified);
CREATE INDEX moz_bookmarks_dateaddedindex ON moz_bookmarks (dateAdded);
CREATE UNIQUE INDEX moz_bookmarks_guid_uniqueindex ON moz_bookmarks (guid);
CREATE UNIQUE INDEX moz_keywords_placepostdata_uniqueindex ON moz_keywords (place_id, post_data);
CREATE UNIQUE INDEX moz_annos_placeattributeindex ON moz_annos (place_id, anno_attribute_id);
CREATE UNIQUE INDEX moz_items_annos_itemattributeindex ON moz_items_annos (item_id, anno_attribute_id);

Upvotes: 1

Pierre-Antoine LaFayette
Pierre-Antoine LaFayette

Reputation: 24402

The Firefox SQLite Manager Addon is a great tool. If you wish to learn about the Firefox Places design and DB schema visit Mozilla Places.

Upvotes: 1

Vitor Silva
Vitor Silva

Reputation: 17612

I also found the following links to be interesting:

After adding a reference to System.Data.Sqlite in my .Net project, all I had to do to create a connection was:

cnn = New SQLiteConnection("data source=c:\Users\user.name\AppData\Roaming\Mozilla\Firefox\Profiles\646vwtnu.default\places.sqlite")
cnn.Open()

I had one minor glitch has the .net sqlite provider does not support sqlite3_enable_shared_cache which I believe is preventing me to open the places.sqlite database while having firefox running (see Support for sqlite3_enable_shared_cache)

Upvotes: 3

hasseg
hasseg

Reputation: 6807

I believe places.sqlite is the one you should be looking into for history (Firefox 3). Below are a couple of Mozilla wiki entries that have some info on the subject.

In earlier versions of Firefox they stored history in a file called history.dat, which was encoded in a format called "Mork". This perl script by Jamie Zawinski can be used to parse Mork files.

Upvotes: 6

Related Questions