PsyPhi Group urn:uuid:

Scaling platform:<br /><br />88m global uniques 1.5m blogs 215m pageviews/day<br /><br /> - 7 boxes, $1500/mo.<br /><br /> - 2 balancers, 2Gb memory, any disk, pound+wackamole+spread.<br /><br /> - 2 databases, 4Gb+ memory, fast disk (RAID), master+slave mysql, split read/write.<br /><br /> - 3 webs, fast cpu, 2Gb memory, litespeed or well-configured apache.<br /><br /> - Everything in subversion.<br /><br /> - Be stateless (shared nothing)<br /><br /> - Memcached<br /><br />geo-targetting of dns, might as well use a CDN<br /><br />single box 300 req/sec 29.5m/day<br /><br />Scaling community:<br /><br />Ref: sxsw presentation<br /><br />Scaling business:<br /><br />Tie revenue streams to cost scaling, e.g. pageviews rather than bloggers. Ads scale revenues with pageviews, effective CPM, low outlay. VIP class of users with advanced features, customisation etc.<br /><br />Scaling people:<br /><br />Hire people as good as, or better than you. Great people = Rich environment + worthwhile problems. 5 things to look for:<br /><br /> - Personality fit (when the shit hits the fan)<br /><br /> - Ability to learn (curiosity)<br /><br /> - Taste (can't be taught)<br /><br /> - Passion for the space<br /><br /> - Familiarity with current technologies<br /><br /> Don't put out too easily - don't hire if doubts exists<br /><br />It's fun to know that the folks at wordpress manage deployment with ssh scripts running svn up.<br /><br />I disagree with Matt's QA response suggesting that you should never do your own hosting. In principle that's ok for the large faceless mass of current web sites & traffic but under special circumstances it's the only way. I build web apps on top of bioinformatics pipelines supporting (today) 320Tb data and there's no way on Earth this would be possible to outsource.

2007-10-03T15:18:05Z urn:uuid:12c6fc06c99a462375eeb3f43dfd832b08ca9e17

Dave's engaging and talking fast so more commentary and less transcription this time. Facebook is an impressive example of social networking success. Apparently incredible growth, doubling every 6 months with near 50% users returning every day.<br /><br />Facebook is about enabling mapping of the social graph. The power of the social graph enables near viral transfer of functionality and features. Facebook photos has nearly twice the traffic of all the other photo sites combined.<br /><br />The Facebook platform aims to allow developers to create applications which leverage and add context to the social graph via deep integration with the platform. Like so many other apps presenting here, it all hinges on open APIs and transparent, open data access.<br /><br />The Facebook opportunity chain looks something like this: Innovation, growth, engagement, monetisation.

2007-10-04T11:57:50Z urn:uuid:632667547e7cd3e0466547863e1207a8c0c0c549

For some time now at <a href="http://www.sanger.ac.uk/">Sanger</a> we've been looking at the problems and solutions involved with building services supporting what are likely to become some of the biggest databases on the planet. The biggest problem is there aren't too many people doing this kind of thing and who are willing to talk about it.<br /><br />The data we're storing falls into two categories. Short Read Format (SRF) files containing sequence, quality and trace (~10Gb per lane) data and FastQ containing sequence and quality (~1Gb per lane).<br /><br />Our requirements for these data are fundamentally for two different systems. One is a long-term archival system for SRF, the responsibility for which will eventually be shifted to the <a href="http://www.ebi.ac.uk/">EBI</a> . The second is, for me at least, the more interesting system -<br /><br />The short-term storage of reads and qualities (and possibly also for selected alignments) isn't the biggest problem - that honour is left to the fast, parallel retrieval of the same. The underlying data store needs to grow at a respectable 12TB per year and serve maybe a hundred simultaneous users requesting up to 1000 sequences per second.<br /><br />Transfer times for reads are small but as a result are disproportionately affected by artefacts like TCP setup times, HTTP header payloads and certainly index seek times.<br /><br />We're looking at a few horizontally-scaling solutions for performing these kinds of jobs - the most obvious are tools like <a href="http://labs.google.com/papers/mapreduce.html">MapReduce</a> and equivalents like <a href="http://hadoop.apache.org/core/">Hadoop</a> running with <a href="http://lucene.apache.org/nutch/">Nutch</a> . My personal favourite and the one I'm holding out for is <a href="http://www.danga.com/mogilefs/">MogileFS</a> from the same people who brought you <a href="http://www.danga.com/memcached/">Memcached</a> . Time to get benchmarking!<br /><br />Updated: Loved <a href="http://teddziuba.com/2008/04/im-going-to-scale-my-foot-up-y.html">this</a> via <a href="http://brad.livejournal.com/">Brad</a>

2008-04-30T00:02:59Z urn:uuid:c5b76da3e608d34edb07244cd9b875ee86906328