B2B Marketplace
Django · PostgreSQL · GIN
SEO · Webmaster API · 11M records
B2B marketplace: SEO infrastructure and search for a database of 11M+ companies
A nationwide industrial B2B platform: national company registry data, 580+ regional subdomains, 1,701 sites verified via the Webmaster API. Search down from 15 seconds to 10 ms — on the full dataset.
11.4 M
companies from the national registry in the platform's database
10 ms
search time across 400K products after GIN
1,701 sites
subdomains verified via the Webmaster API
1,187 files
sitemap files, 1.86 GB, full crawl coverage
Context
A nationwide industrial B2B platform: 11.4M companies from the national registry, 398K products, 580+ city and regional subdomains. Stack: Django + PostgreSQL, Celery, OAuth, Webmaster API.
Django · PostgreSQL
Registry data 11.4M
IP registration
Problem
ORM search across 400K products took 8–15 seconds: unacceptable. 580+ subdomains weren't verified with the search engine's webmaster tools — organic traffic was blocked. Duplicate content without canonical tags → SEO penalty.
8–15s search
0 verified domains
canonical missing
Platform scale
The numbers behind the infrastructure we worked with
Each figure is a separate engineering challenge: import, storage, indexing, crawling
Registry companies
11,400,000
records · bulk import · PostgreSQL
Products
398,000
items · GIN trigram index
PersonProfile
7,500,000
profiles · claim/verification flow
Subdomains
580+
city and regional websites
Sitemap files
1,187
files · 1.86 GB · static generation
Webmaster API
1,701
sites verified via the API
Response time · search across 400K products
GIN trigram index: seq scan → index scan · ×1200 speedup
# products/views.py — slow
def search_products(query):
# ❌ ORM → LIKE '%query%'
# ❌ seq scan across 400K rows
return Product.objects\
.filter(
name__icontains=query
)\
.select_related('company')
# → 8–15 seconds in prod
# products/search.py — fast
from django.db import connection
def search_products(query):
with connection.cursor() as cur:
cur.execute("""
SELECT id, name, company_id
FROM products_product
WHERE name ILIKE %s
LIMIT 50
""", [f'%{query}%'])
return cur.fetchall()
# → ~10 ms thanks to the GIN index
-- Enable the pg_trgm extension
CREATE EXTENSION IF NOT EXISTS pg_trgm;
-- GIN index on the name column
CREATE INDEX CONCURRENTLY
idx_product_name_trgm
ON products_product
USING GIN (name gin_trgm_ops);
-- EXPLAIN ANALYZE confirms:
-- Seq Scan → Bitmap Index Scan
-- cost: 24800 → 18 rows: 400K → 50
# 580+ subdomains like:
# city-a.platform.example
# city-b.platform.example
# region-c.platform.example
# ❌ none of them verified
# ❌ search engines don't index them
# ❌ organic traffic = 0
# manual verification: ~3 min/site
# × 1,701 = ~85 hours of work
# webmaster_api.py
import requests
def verify_all_subdomains(token, domains):
session = requests.Session()
session.headers['Authorization'] = \
f'OAuth {token}'
verified = 0
for domain in domains:
r = session.post(
f"{API}/user/{USER}/hosts",
json={"name": domain}
)
if r.status_code == 200:
verified += 1
return verified # → 1,701
# Generation in batches of 50K URLs
CHUNK = 50_000
def generate_sitemaps(domain, qs):
total = qs.count()
files = []
for i, offset in enumerate(
range(0, total, CHUNK)
):
chunk = qs[offset:offset+CHUNK]
fname = f"sitemap-{domain}-{i}.xml"
write_xml(fname, chunk)
files.append(fname)
write_index(f"sitemap-{domain}.xml", files)
return len(files) # → 1,187 total
{# base.html — eliminating duplicates #}
<!-- Without canonical: 580 copies -->
{# ❌ every subdomain = a duplicate #}
{# search engines penalize duplicates #}
<!-- With canonical: one source -->
<link rel="canonical"
href="https://platform.example
{{ request.path }}">
{# ✓ all regions → the main domain #}
{# ✓ SEO weight consolidated #}
{# ✓ duplication eliminated #}
Solutions delivered:
✓ GIN trigram index
✓ ×1200 search
✓ 1,701 sites verified
✓ canonical tags
✓ 1,187 sitemap files
Result
Search runs in real time on a database of 11.4M records. All regional subdomains are verified and open for indexing. 1,187 sitemap files provide full crawl coverage. The platform is registered as software with the national IP office.
Django
PostgreSQL · GIN trigram
pg_trgm
Webmaster API
Company registry · bulk import
Celery
IP registration · software