Import Wikipedia into MySQL with MediaWiki

A Wikipedia dump can live in MySQL or MariaDB, but it is not a SQL backup that can be loaded into an arbitrary database. The XML describes MediaWiki pages and revisions. MediaWiki must own the schema and perform the import so that revision records, content models, actor identities, and derived tables remain consistent.

That distinction matters because the old shortcut - converting XML with MWDumper and piping its SQL into MySQL - no longer works for current dumps. MWDumper is archived, targets the MediaWiki 1.25 schema, and cannot process dumps generated by MediaWiki 1.31 or later. The supported general route is an installed MediaWiki instance backed by MySQL or MariaDB, followed by MediaWiki's importDump maintenance command.

What a Wikipedia Dump Contains

Wikimedia publishes several products for each wiki. Pick the smallest one that preserves the data your project actually needs:

• pages-articles.xml.bz2 contains the current revision of content pages, templates, modules, and redirects. It excludes discussion and user pages and is the practical choice for a searchable local corpus.
• pages-meta-current.xml.bz2 contains the current revision of pages in all namespaces.
• pages-meta-history contains revision history and is split into many very large files. Use it only when historical revisions are part of the analysis.

None of these is a complete clone of Wikipedia. XML content dumps do not include image binaries, user accounts, most logs, search indexes, or every auxiliary database table. Wikimedia also publishes selected tables as SQL dumps, but those table snapshots are separate datasets; they do not replace the MediaWiki content import.

Prepare MediaWiki and MySQL

Start with a supported MediaWiki installation configured for MySQL or MariaDB. Complete the installer before importing. It creates the schema that exactly matches the installed MediaWiki release and writes the database connection to LocalSettings.php. Do not download a standalone tables.sql from another release: MediaWiki's schema changes over time.

A dedicated empty wiki is the safest target. importDump can resume an interrupted run and skips revisions already present, but importing into a populated wiki may merge page histories and preserve newer local revisions as the visible version. Take a database backup before importing into anything valuable.

Capacity planning is not optional. Check the current compressed size in the dump index, reserve additional space for decompression, InnoDB data, indexes, and temporary work, and expect a large language edition to run for days or weeks. The complete English article dump alone is measured in tens of gigabytes while compressed.

Download and Verify the Dump

The latest directory is convenient for a one-off import. For a reproducible pipeline, open the Wikimedia dump index, select a completed dated snapshot, and pin that date in both the URL and your records. This example uses the current English article dump:

mkdir -p /srv/wikimedia-dumps/enwiki
cd /srv/wikimedia-dumps/enwiki

wget -c https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2
wget https://dumps.wikimedia.org/enwiki/latest/sha1sums

Verify the compressed file before spending days importing it. On GNU/Linux, select the matching line from Wikimedia's checksum manifest and pass it to sha1sum:

grep ' enwiki-latest-pages-articles.xml.bz2$' sha1sums | sha1sum --check -

A successful check prints OK. Keep the compressed file: MediaWiki recognizes the .bz2 suffix and decompresses it while reading, so a second uncompressed copy is not required.

Validate Before Writing

Run all maintenance commands from the directory containing LocalSettings.php. Since MediaWiki 1.40, maintenance/run.php is the stable entry point. A dry run parses the dump and catches unreadable input or XML-format errors without changing the database:

cd /var/www/mediawiki

php maintenance/run.php importDump \
  --conf ./LocalSettings.php \
  --dry-run \
  --report 1000 \
  /srv/wikimedia-dumps/enwiki/enwiki-latest-pages-articles.xml.bz2

A full dry run still reads the entire archive. For a very large edition, first prove the setup with one of the smaller numbered dump parts listed in the same directory, then schedule the real import in tmux, screen, or a supervised batch job.

Import Wikipedia

Remove --dry-run to write through MediaWiki into MySQL:

cd /var/www/mediawiki

php maintenance/run.php importDump \
  --conf ./LocalSettings.php \
  --report 1000 \
  /srv/wikimedia-dumps/enwiki/enwiki-latest-pages-articles.xml.bz2

Progress reports show page and revision throughput. If the process is interrupted, run the same command again; the importer skips revisions already loaded. This is safer than translating the stream to SQL because a failing SQL pipeline may continue after an error and leave an incomplete schema without a reliable resume point.

For a faster bulk load, --no-updates postpones derived link-table work:

php maintenance/run.php importDump \
  --conf ./LocalSettings.php \
  --no-updates \
  --report 1000 \
  /srv/wikimedia-dumps/enwiki/enwiki-latest-pages-articles.xml.bz2

This deliberately leaves links, templates, and categories inconsistent until the rebuild step. Do not expose the wiki as complete between those two operations. For the English Wikipedia or a full history import, benchmark a numbered part first; the general importer prioritizes correctness over maximum bulk-loading speed.

Rebuild Derived Data

After an import with --no-updates, rebuild links and parser-derived tables. Then update site statistics, which importDump does not maintain, and optionally reconstruct recent changes:

cd /var/www/mediawiki

php maintenance/run.php rebuildall --conf ./LocalSettings.php
php maintenance/run.php initSiteStats --conf ./LocalSettings.php --update
php maintenance/run.php rebuildrecentchanges --conf ./LocalSettings.php

Run MediaWiki's normal job queue as well. Extensions may add their own maintenance tasks, and a production search service such as OpenSearch needs a separate index build. Imported templates can also depend on extensions and configuration used by Wikimedia; successful database insertion does not guarantee identical rendering.

Verify the Result

Check both MediaWiki and MySQL instead of treating a zero exit status as the only acceptance test:

SELECT COUNT(*) AS pages FROM page;
SELECT COUNT(*) AS revisions FROM revision;
SELECT page_namespace, COUNT(*) AS pages
FROM page
GROUP BY page_namespace
ORDER BY page_namespace;

Compare these counts with Special:Statistics, open representative articles, templates, modules, and redirects, and inspect the import log for warnings. Missing images are expected because the article dump contains file-description pages, not the media binaries themselves.

Using MySQL for Analysis

MediaWiki's tables are an application schema, not a stable analytics API. Querying them directly is useful for local exploration, but bind every query to the schema documentation for your installed release. Text, slots, content, revisions, actors, and comments are normalized across several tables; assumptions based on old page, revision, and text joins are often wrong on current releases.

For repeated analysis, leave the imported wiki as the source of truth and materialize a separate, versioned analytics schema. That isolates long scans from the application, lets you denormalize only the fields you need, and prevents a later MediaWiki upgrade from silently changing query semantics.

References

• [Dumps]Wikimedia Downloads: Database backup dumps.
• [Import]MediaWiki Manual: importDump.php.
• [XML]MediaWiki Manual: Importing XML dumps.
• [MWDumper]MediaWiki Manual: MWDumper (archived and obsolete).