OMFGZBLOGS!!!1
(Blogging & Journaling)

Brad Whitaker – Six Apart / LiveJournal

Frank Precissi - DeadJournal

Who We Are

Brad Whitaker

Software Engineer, Six Apart / LiveJournal

Computer Science, University of Cincinnati

6 years with LiveJournal

Frank Precissi

System Administrator/Founder, DeadJournal

Doctorate of Pharmacy, University of the Pacific

4 years with DeadJournal

Blogging vs Journaling

Blogging

Intent is to publish to the public

Limited social networking

Limited entry security

Journaling

Intent is to publish to social network (friends)

Ability to restrict entries to certain groups or individuals

Less “journalistic” (or something like that)

What is LiveJournal?

Open-Source journaling platform

livejournal.com: 7+ million users, 5,000+ hits/sec peak

deadjournal.com: 500+ thousand users, way less/sec!

Features

Scalability

Single Server to 100's of servers

Built in aggregator (RSS, Atom, LJ Users)

Threaded comments

Fine grained account security

Private, Public, Friends, Groups of Friends

Extensive community support

Moderation, Maintainers, Controlled Access

Technical Overview

Software

Linux (Prefer Debian)

Support Open Source and Free Software

Apache/mod_perl (no Apache2/mod_perl2 port yet)

Fast, mature, stable.

MySQL

Fast, native replication

Perl

Mature of technologies available when project was started in 1998. Personal preference/experience. We're not zealots.

Hardware

32bit webslaves

Master-Master 64bit user database partitions (clusters)

JBOD arrays (mogilefs)

Big/IP Load Balancer

Obstacles in Reaching 1+ Million Accounts
“Scale Out, Not Up”

Web Nodes

CPU utilization

mod_perl + lots of hardware

Effective load balancing

Big/IP + In-House software load balancing (Perlbal)

Databases

Connection Limits

No persistent connections with MySQL, partitioning

Concurrency/Locking

<3 InnoDB

I/O: Disks are slow

Memcached

File Storage

MogileFS on JBOD arrays

Database Partitioning
MySQL Replication & DBI::Role

2 Tiers

Global Master & Slaves

User Master-Master clusters

Current master & hot-spare

DBI::Role: Our library on top of Perl's DBI

Returns database handles based on role

LJ::get_dbh(“cluster10a”), LJ::get_dbh(“master”), LJ::get_dbh(“slow”), etc

Dynamic weighting of busy/downed databases

Load Balancing: Perlbal
(http://www.danga.com/perlbal)

All-Perl epoll/kqueue based load balancer

Designed in-house to provide fine-grained load balancing that was not possible with a big/ip

Acts as both a load balancer and reverse proxy

Uses sendfile() to handle static content without contacting CPU-expensive mod_perl processes

Able to adjust settings on-the-fly

Tight integration with MogileFS (more later)

Easily extensible plugin architecture

Perlbal: Request Cycle

Distributed In-Memory Object Caching: memcached
(http://www.danga.com/memcached)

Lightweight daemon written in C using epoll

Distributed in-memory hash table

key/value pairs hashed out to memcached nodes on network

each memcached node is a hash bucket

multiple nodes (memcached procs) per host common

Memory fragmentation is internal

slab allocation mechanism

Stores arbitrary data (max 1 megabyte currently)

storable, pickle, gzip, etc

api for perl/python/php/C/etc

Used by large sites (wikipedia/slashdot/etc)

Memcached (continued)

Simple and lightweight protocol

SET <key> <value> <lifetime>

GET <key>

etc

Why not use query cache in MySQL 4?

Still requires a database query to hit cache

memcached was developed to reduce database hits on static content like username/password pairs, rendered journal styles

relevant data in one place, no query parsing, tying up MySQL thread, etc

Unable to specify cached object lifetime

Why not cache on each webslave?

Far too local, low hit rates.

~90-93% actual cache hit rate for LiveJournal

Distributed Storage: MogileFS
(http://www.danga.com/mogilefs)

Not actually a block device or filesystem

Tracker-based file-spraying / retrieval system

3 Components

MySQL-backed tracker(s)

Mogstored: Perlbal-based storage engine

Application APIs

Each file lives on N hosts (configurable per-class)

Some files can easily be recreated, so redundancy is wasteful. Eg: image thumbnails.

Application talks to tracker, which responds with Perlbal-based storage node to contact via HTTP

The Big Picture: Large File Request

On the Horizon

OpenID (http://www.danga.com/openid)

Decentralized cross-site identification scheme

Users “home site” vouches for their identity elsewhere

Eg: DeadJournal users can comment on LiveJournal without having a LiveJournal account

Tag support on LiveJournal

Users can tag their entries and browse/search later

Currently it is per user, not global

FotoBilder (http://pics.livejournal.com)

“LiveJournal ScrapBook”

Full featured photohosting and sharing

Tight LiveJournal integration

Questions?

PLUG: LiveJournal is hiring talented software Engineers & Interns!
http://www.sixapart.com/about/jobs

2005-06-05T17:08:16